[gpaw-users] gpaw-users Digest, Vol 58, Issue 15
Toma Susi
toma.susi at univie.ac.at
Tue Nov 18 10:36:08 CET 2014
I'm no expert on this, but I'd say you are better off testing parallelization with the number of cores (and k-points) you want to use.
Dr. Toma Susi
FWF Lise Meitner fellow
University of Vienna, Austria
http://mostlyphysics.wordpress.com
"Love is the only emotion that enhances our intelligence."
-Humberto Maturana
> On 18 Nov 2014, at 10:24, Trevor Hardcastle <T.P.Hardcastle at leeds.ac.uk> wrote:
>
> Hi Toma,
>
> I was using only 16/32 cores for testing purposes. I expect to be using up to ~512 cores for my larger systems once I've got to grips with effective parallelisation, so scalapack will certainly be a priority for me. Perhaps I'm better off testing with larger numbers of cores?
>
> Thanks,
>
> Trevor
>
> -----Original Message-----
> From: Toma Susi [mailto:toma.susi at univie.ac.at]
> Sent: 18 November 2014 07:34
> To: Trevor Hardcastle
> Cc: gpaw-users at listserv.fysik.dtu.dk
> Subject: Re: gpaw-users Digest, Vol 58, Issue 15
>
> Hi Trevor,
>
> I would just run without Scalapack; with so few processors it is not necessary but makes parallelization a pain in the neck (+ weird errors).
>
> Cheers,
> Toma
>
>
>
>> On 17.11.2014, at 20.34, gpaw-users-request at listserv.fysik.dtu.dk wrote:
>>
>> Send gpaw-users mailing list submissions to
>> gpaw-users at listserv.fysik.dtu.dk
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>> or, via email, send a message with subject or body 'help' to
>> gpaw-users-request at listserv.fysik.dtu.dk
>>
>> You can reach the person managing the list at
>> gpaw-users-owner at listserv.fysik.dtu.dk
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of gpaw-users digest..."
>>
>>
>> Today's Topics:
>>
>> 1. Remember IRC-meeing tomorrow (Jens J?rgen Mortensen)
>> 2. diagonalize_full_hamiltonian: Assertion error assert
>> bd.comm.size == nprow * npcol (Trevor Hardcastle)
>> 3. Re: diagonalize_full_hamiltonian: Assertion error assert
>> bd.comm.size == nprow * npcol (Thomas Stenb?k Jauho)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Mon, 17 Nov 2014 16:19:13 +0100
>> From: Jens J?rgen Mortensen <jensj at fysik.dtu.dk>
>> To: gpaw-users <gpaw-users at listserv.fysik.dtu.dk>
>> Subject: [gpaw-users] Remember IRC-meeing tomorrow
>> Message-ID: <546A11F1.2060700 at fysik.dtu.dk>
>> Content-Type: text/plain; charset="utf-8"; format=flowed
>>
>> 15:00 danish time: #gpaw on frnode.net
>>
>> Jens J?rgen
>>
>>
>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Mon, 17 Nov 2014 19:16:49 +0000
>> From: Trevor Hardcastle <T.P.Hardcastle at leeds.ac.uk>
>> To: "gpaw-users at listserv.fysik.dtu.dk"
>> <gpaw-users at listserv.fysik.dtu.dk>
>> Subject: [gpaw-users] diagonalize_full_hamiltonian: Assertion error
>> assert bd.comm.size == nprow * npcol
>> Message-ID:
>>
>> <0FF15BFDE4634440B5F0D983CDF44C4E019B08E15C56 at HERMES8.ds.leeds.ac.uk>
>> Content-Type: text/plain; charset="us-ascii"
>>
>> Hi all,
>>
>> Brand new GPAW! user here. I'm trying to get a basic EEL spectrum for N-doped graphene using gpaw/0.11.0.12151. The calculation works fine when using 16 cores, but fails when I try with 32 cores. The exact same .py script was used in both cases:
>>
>>
>> **********************************************************************
>> *********************************
>> from __future__ import print_function
>> import numpy as np
>> from ase import Atoms
>> from ase.parallel import paropen
>> from gpaw import GPAW, FermiDirac, PW
>> from gpaw.response.df import DielectricFunction
>>
>> graphene = Atoms('C7N',
>>
>> scaled_positions = [(0.8333333333333334, 0.1666666666666666, 0.5000000000000000),
>> (0.8333333333333334, 0.6666666666666669, 0.5000000000000000),
>> (0.3333333333333332, 0.1666666666666666, 0.5000000000000000),
>> (0.1666666666666663, 0.3333333333333329, 0.5000000000000000),
>> (0.6666666666666672, 0.8333333333333337, 0.5000000000000000),
>> (0.1666666666666664, 0.8333333333333339, 0.5000000000000000),
>> (0.3333333333333334, 0.6666666666666667, 0.5000000000000000),
>> (0.6666666666666667, 0.3333333333333334, 0.5000000000000000)],
>>
>> pbc=(1, 1, 1),
>>
>> cell=[(4.260844986619441, -2.460000000000003, 0.000000000000000),
>> (-0.000000000000000, 4.920000000000004, 0.000000000000000),
>> (0.000000000000000, 0.000000000000000, 10.000000000000000)]
>> )
>>
>>
>> calc = GPAW(mode=PW(100),
>> nbands=60,
>> basis='dzp',
>> eigensolver='rmm-diis',
>> xc='LDA',
>> kpts=(5, 5, 1),
>> txt='graphene_1.txt')
>>
>> graphene.set_calculator(calc)
>> graphene.get_potential_energy()
>>
>> calc.set(kpts=(20, 20, 1), fixdensity=True)
>> calc.set(fixdensity=True)
>> graphene.get_potential_energy()
>>
>> calc.diagonalize_full_hamiltonian(nbands=60)
>> calc.write('graphene.gpw', 'all')
>>
>> df = DielectricFunction(calc='graphene.gpw',
>> domega0=0.01,
>> eta=0.3,
>> ecut=20,
>> txt='G_to_G.txt')
>>
>> q_c = [0.0, 0.0, 0.0]
>>
>> df.get_eels_spectrum(q_c=q_c, filename='G_to_G_EELS.csv')
>> **********************************************************************
>> *********************************
>>
>> When I run this on 32 cores, I get these assertion errors:
>>
>>
>> * AssertionError
>> calc.diagonalize_full_hamiltonian(nbands=60)
>> "assert bd.comm.size == nprow * npcol" on line 746 of
>> $HOME/gpaw/0.11.0.12151/lib/python2.7/site-packages/gpaw/wavefunctions
>> /pw.py
>>
>>
>> * GPAW CLEANUP (node 23): <type 'exceptions.AssertionError'> occurred. Calling MPI_Abort!
>> File "/home/ufaserv1_g/pretha/soft/gpaw/0.11.0.12151/lib/python2.7/site-packages/gpaw/paw.py", line 1000, in diagonalize_full_hamiltonian
>> nbands, scalapack)
>>
>>
>> What exactly are the quantities "bd.comm.size", "nprow" and "npcol",
>> and how can I avoid encountering these errors? I'm using the ARC2
>> machine at Leeds University:
>> https://hec.wiki.leeds.ac.uk/bin/view/Documentation/WebHome
>>
>> Many thanks in advance for any tips.
>>
>> Trevor
>>
>> ************************
>> Dr. Trevor P. Hardcastle
>> EPSRC Doctoral Prize Fellow
>> Room B20.C
>> Engineering Building
>> Institute for Materials Research
>> University of Leeds
>> LS2 9JT, United Kingdom
>> t.p.hardcastle at leeds.ac.uk<mailto:t.p.hardcastle at leeds.ac.uk>
>> ************************
>>
>> -------------- next part -------------- An HTML attachment was
>> scrubbed...
>> URL:
>> <http://listserv.fysik.dtu.dk/pipermail/gpaw-users/attachments/2014111
>> 7/ab7f2ba5/attachment-0001.html>
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Mon, 17 Nov 2014 11:33:58 -0800
>> From: Thomas Stenb?k Jauho <s093006 at student.dtu.dk>
>> To: gpaw-users at listserv.fysik.dtu.dk
>> Subject: Re: [gpaw-users] diagonalize_full_hamiltonian: Assertion
>> error assert bd.comm.size == nprow * npcol
>> Message-ID: <d7cdd43ee4334a60cb9aa0816d358aa6 at student.dtu.dk>
>> Content-Type: text/plain; charset=UTF-8; format=flowed
>>
>> Hi Trevor,
>>
>> To me it seems like an issue with scalapack and the parallelization
>> you are trying.
>>
>> As I understand the code will try to parallelize over bands when using
>> 32 cores (as it cannot parallelize 5x5x1 = 25 kpts on 32 cores).
>>
>> Therefore it will switch to band parallelization, however in order to
>> do this successfully you should set scalapack correspondingly.
>>
>> As an example I use the following code for parallelizing the
>> diagonalization over 8 or 16 cores respectively:
>>
>> calc.diagonalize_full_hamiltonian(nbands=nbands, scalapack=(4,4,32))
>>
>> This means nprow=4 and npcol=4, their product should equal the number
>> of cores you're using, unless spin parallelization is included, in
>> which case the product needs to be half the number of cores.
>>
>> Best,
>> Thomas
>>
>>
>>
>>
>>> On 2014-11-17 11:16, Trevor Hardcastle wrote:
>>> Hi all,
>>>
>>> Brand new GPAW! user here. I'm trying to get a basic EEL spectrum for
>>> N-doped graphene using gpaw/0.11.0.12151. The calculation works fine
>>> when using 16 cores, but fails when I try with 32 cores. The exact
>>> same .py script was used in both cases:
>>>
>>> *********************************************************************
>>> **********************************
>>>
>>> from __future__ import print_function
>>>
>>> import numpy as np
>>>
>>> from ase import Atoms
>>>
>>> from ase.parallel import paropen
>>>
>>> from gpaw import GPAW, FermiDirac, PW
>>>
>>> from gpaw.response.df import DielectricFunction
>>>
>>> graphene = Atoms('C7N',
>>>
>>> scaled_positions = [(0.8333333333333334, 0.1666666666666666,
>>> 0.5000000000000000),
>>>
>>> (0.8333333333333334, 0.6666666666666669, 0.5000000000000000),
>>>
>>> (0.3333333333333332, 0.1666666666666666, 0.5000000000000000),
>>>
>>> (0.1666666666666663, 0.3333333333333329, 0.5000000000000000),
>>>
>>> (0.6666666666666672, 0.8333333333333337, 0.5000000000000000),
>>>
>>> (0.1666666666666664, 0.8333333333333339, 0.5000000000000000),
>>>
>>> (0.3333333333333334, 0.6666666666666667, 0.5000000000000000),
>>>
>>> (0.6666666666666667, 0.3333333333333334, 0.5000000000000000)],
>>>
>>> pbc=(1, 1, 1),
>>>
>>> cell=[(4.260844986619441, -2.460000000000003, 0.000000000000000),
>>>
>>> (-0.000000000000000, 4.920000000000004, 0.000000000000000),
>>>
>>> (0.000000000000000, 0.000000000000000, 10.000000000000000)]
>>>
>>> )
>>>
>>> calc = GPAW(mode=PW(100),
>>>
>>> nbands=60,
>>>
>>> basis='dzp',
>>>
>>> eigensolver='rmm-diis',
>>>
>>> xc='LDA',
>>>
>>> kpts=(5, 5, 1),
>>>
>>> txt='graphene_1.txt')
>>>
>>> graphene.set_calculator(calc)
>>>
>>> graphene.get_potential_energy()
>>>
>>> calc.set(kpts=(20, 20, 1), fixdensity=True)
>>>
>>> calc.set(fixdensity=True)
>>>
>>> graphene.get_potential_energy()
>>>
>>> calc.diagonalize_full_hamiltonian(nbands=60)
>>>
>>> calc.write('graphene.gpw', 'all')
>>>
>>> df = DielectricFunction(calc='graphene.gpw',
>>>
>>> domega0=0.01,
>>>
>>> eta=0.3,
>>>
>>> ecut=20,
>>>
>>> txt='G_to_G.txt')
>>>
>>> q_c = [0.0, 0.0, 0.0]
>>>
>>> df.get_eels_spectrum(q_c=q_c, filename='G_to_G_EELS.csv')
>>>
>>> *********************************************************************
>>> **********************************
>>>
>>> When I run this on 32 cores, I get these assertion errors:
>>>
>>> ? AssertionError
>>>
>>> calc.diagonalize_full_hamiltonian(nbands=60)
>>>
>>> "assert bd.comm.size == nprow * npcol" on line 746 of
>>> $HOME/gpaw/0.11.0.12151/lib/python2.7/site-packages/gpaw/wavefunction
>>> s/pw.py
>>>
>>> ? GPAW CLEANUP (node 23): <type 'exceptions.AssertionError'>
>>> occurred. Calling MPI_Abort!
>>>
>>> File
>>> "/home/ufaserv1_g/pretha/soft/gpaw/0.11.0.12151/lib/python2.7/site-pa
>>> ckages/gpaw/paw.py", line 1000, in diagonalize_full_hamiltonian
>>>
>>> nbands, scalapack)
>>>
>>> What exactly are the quantities "bd.comm.size", "nprow" and "npcol",
>>> and how can I avoid encountering these errors? I'm using the ARC2
>>> machine at Leeds University:
>>> https://hec.wiki.leeds.ac.uk/bin/view/Documentation/WebHome [1]
>>>
>>> Many thanks in advance for any tips.
>>>
>>> Trevor
>>>
>>> ************************
>>>
>>> Dr. Trevor P. Hardcastle
>>>
>>> EPSRC Doctoral Prize Fellow
>>>
>>> Room B20.C
>>>
>>> Engineering Building
>>>
>>> Institute for Materials Research
>>>
>>> University of Leeds
>>>
>>> LS2 9JT, United Kingdom
>>>
>>> t.p.hardcastle at leeds.ac.uk
>>>
>>> ************************
>>>
>>>
>>>
>>> Links:
>>> ------
>>> [1] https://hec.wiki.leeds.ac.uk/bin/view/Documentation/WebHome
>>>
>>> _______________________________________________
>>> gpaw-users mailing list
>>> gpaw-users at listserv.fysik.dtu.dk
>>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>>
>>
>> ------------------------------
>>
>> _______________________________________________
>> gpaw-users mailing list
>> gpaw-users at listserv.fysik.dtu.dk
>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>>
>> End of gpaw-users Digest, Vol 58, Issue 15
>> ******************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.fysik.dtu.dk/pipermail/gpaw-users/attachments/20141118/173e5891/attachment-0001.html>
More information about the gpaw-users
mailing list