[gpaw-users] gpaw-users Digest, Vol 58, Issue 15
Toma Susi
toma.susi at univie.ac.at
Tue Nov 18 08:33:47 CET 2014
Hi Trevor,
I would just run without Scalapack; with so few processors it is not necessary but makes parallelization a pain in the neck (+ weird errors).
Cheers,
Toma
> On 17.11.2014, at 20.34, gpaw-users-request at listserv.fysik.dtu.dk wrote:
>
> Send gpaw-users mailing list submissions to
> gpaw-users at listserv.fysik.dtu.dk
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
> or, via email, send a message with subject or body 'help' to
> gpaw-users-request at listserv.fysik.dtu.dk
>
> You can reach the person managing the list at
> gpaw-users-owner at listserv.fysik.dtu.dk
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of gpaw-users digest..."
>
>
> Today's Topics:
>
> 1. Remember IRC-meeing tomorrow (Jens J?rgen Mortensen)
> 2. diagonalize_full_hamiltonian: Assertion error assert
> bd.comm.size == nprow * npcol (Trevor Hardcastle)
> 3. Re: diagonalize_full_hamiltonian: Assertion error assert
> bd.comm.size == nprow * npcol (Thomas Stenb?k Jauho)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 17 Nov 2014 16:19:13 +0100
> From: Jens J?rgen Mortensen <jensj at fysik.dtu.dk>
> To: gpaw-users <gpaw-users at listserv.fysik.dtu.dk>
> Subject: [gpaw-users] Remember IRC-meeing tomorrow
> Message-ID: <546A11F1.2060700 at fysik.dtu.dk>
> Content-Type: text/plain; charset="utf-8"; format=flowed
>
> 15:00 danish time: #gpaw on frnode.net
>
> Jens J?rgen
>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 17 Nov 2014 19:16:49 +0000
> From: Trevor Hardcastle <T.P.Hardcastle at leeds.ac.uk>
> To: "gpaw-users at listserv.fysik.dtu.dk"
> <gpaw-users at listserv.fysik.dtu.dk>
> Subject: [gpaw-users] diagonalize_full_hamiltonian: Assertion error
> assert bd.comm.size == nprow * npcol
> Message-ID:
> <0FF15BFDE4634440B5F0D983CDF44C4E019B08E15C56 at HERMES8.ds.leeds.ac.uk>
> Content-Type: text/plain; charset="us-ascii"
>
> Hi all,
>
> Brand new GPAW! user here. I'm trying to get a basic EEL spectrum for N-doped graphene using gpaw/0.11.0.12151. The calculation works fine when using 16 cores, but fails when I try with 32 cores. The exact same .py script was used in both cases:
>
>
> *******************************************************************************************************
> from __future__ import print_function
> import numpy as np
> from ase import Atoms
> from ase.parallel import paropen
> from gpaw import GPAW, FermiDirac, PW
> from gpaw.response.df import DielectricFunction
>
> graphene = Atoms('C7N',
>
> scaled_positions = [(0.8333333333333334, 0.1666666666666666, 0.5000000000000000),
> (0.8333333333333334, 0.6666666666666669, 0.5000000000000000),
> (0.3333333333333332, 0.1666666666666666, 0.5000000000000000),
> (0.1666666666666663, 0.3333333333333329, 0.5000000000000000),
> (0.6666666666666672, 0.8333333333333337, 0.5000000000000000),
> (0.1666666666666664, 0.8333333333333339, 0.5000000000000000),
> (0.3333333333333334, 0.6666666666666667, 0.5000000000000000),
> (0.6666666666666667, 0.3333333333333334, 0.5000000000000000)],
>
> pbc=(1, 1, 1),
>
> cell=[(4.260844986619441, -2.460000000000003, 0.000000000000000),
> (-0.000000000000000, 4.920000000000004, 0.000000000000000),
> (0.000000000000000, 0.000000000000000, 10.000000000000000)]
> )
>
>
> calc = GPAW(mode=PW(100),
> nbands=60,
> basis='dzp',
> eigensolver='rmm-diis',
> xc='LDA',
> kpts=(5, 5, 1),
> txt='graphene_1.txt')
>
> graphene.set_calculator(calc)
> graphene.get_potential_energy()
>
> calc.set(kpts=(20, 20, 1), fixdensity=True)
> calc.set(fixdensity=True)
> graphene.get_potential_energy()
>
> calc.diagonalize_full_hamiltonian(nbands=60)
> calc.write('graphene.gpw', 'all')
>
> df = DielectricFunction(calc='graphene.gpw',
> domega0=0.01,
> eta=0.3,
> ecut=20,
> txt='G_to_G.txt')
>
> q_c = [0.0, 0.0, 0.0]
>
> df.get_eels_spectrum(q_c=q_c, filename='G_to_G_EELS.csv')
> *******************************************************************************************************
>
> When I run this on 32 cores, I get these assertion errors:
>
>
> * AssertionError
> calc.diagonalize_full_hamiltonian(nbands=60)
> "assert bd.comm.size == nprow * npcol" on line 746 of $HOME/gpaw/0.11.0.12151/lib/python2.7/site-packages/gpaw/wavefunctions/pw.py
>
>
> * GPAW CLEANUP (node 23): <type 'exceptions.AssertionError'> occurred. Calling MPI_Abort!
> File "/home/ufaserv1_g/pretha/soft/gpaw/0.11.0.12151/lib/python2.7/site-packages/gpaw/paw.py", line 1000, in diagonalize_full_hamiltonian
> nbands, scalapack)
>
>
> What exactly are the quantities "bd.comm.size", "nprow" and "npcol", and how can I avoid encountering these errors? I'm using the ARC2 machine at Leeds University: https://hec.wiki.leeds.ac.uk/bin/view/Documentation/WebHome
>
> Many thanks in advance for any tips.
>
> Trevor
>
> ************************
> Dr. Trevor P. Hardcastle
> EPSRC Doctoral Prize Fellow
> Room B20.C
> Engineering Building
> Institute for Materials Research
> University of Leeds
> LS2 9JT, United Kingdom
> t.p.hardcastle at leeds.ac.uk<mailto:t.p.hardcastle at leeds.ac.uk>
> ************************
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://listserv.fysik.dtu.dk/pipermail/gpaw-users/attachments/20141117/ab7f2ba5/attachment-0001.html>
>
> ------------------------------
>
> Message: 3
> Date: Mon, 17 Nov 2014 11:33:58 -0800
> From: Thomas Stenb?k Jauho <s093006 at student.dtu.dk>
> To: gpaw-users at listserv.fysik.dtu.dk
> Subject: Re: [gpaw-users] diagonalize_full_hamiltonian: Assertion
> error assert bd.comm.size == nprow * npcol
> Message-ID: <d7cdd43ee4334a60cb9aa0816d358aa6 at student.dtu.dk>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
> Hi Trevor,
>
> To me it seems like an issue with scalapack and the parallelization you
> are trying.
>
> As I understand the code will try to parallelize over bands when using
> 32 cores (as it cannot parallelize 5x5x1 = 25 kpts on 32 cores).
>
> Therefore it will switch to band parallelization, however in order to do
> this successfully you should set scalapack correspondingly.
>
> As an example I use the following code for parallelizing the
> diagonalization over 8 or 16 cores respectively:
>
> calc.diagonalize_full_hamiltonian(nbands=nbands, scalapack=(4,4,32))
>
> This means nprow=4 and npcol=4, their product should equal the number of
> cores you're using, unless spin parallelization is included, in which
> case the product needs to be half the number of cores.
>
> Best,
> Thomas
>
>
>
>
>> On 2014-11-17 11:16, Trevor Hardcastle wrote:
>> Hi all,
>>
>> Brand new GPAW! user here. I'm trying to get a basic EEL spectrum for
>> N-doped graphene using gpaw/0.11.0.12151. The calculation works fine
>> when using 16 cores, but fails when I try with 32 cores. The exact
>> same .py script was used in both cases:
>>
>> *******************************************************************************************************
>>
>> from __future__ import print_function
>>
>> import numpy as np
>>
>> from ase import Atoms
>>
>> from ase.parallel import paropen
>>
>> from gpaw import GPAW, FermiDirac, PW
>>
>> from gpaw.response.df import DielectricFunction
>>
>> graphene = Atoms('C7N',
>>
>> scaled_positions = [(0.8333333333333334, 0.1666666666666666,
>> 0.5000000000000000),
>>
>> (0.8333333333333334, 0.6666666666666669, 0.5000000000000000),
>>
>> (0.3333333333333332, 0.1666666666666666, 0.5000000000000000),
>>
>> (0.1666666666666663, 0.3333333333333329, 0.5000000000000000),
>>
>> (0.6666666666666672, 0.8333333333333337, 0.5000000000000000),
>>
>> (0.1666666666666664, 0.8333333333333339, 0.5000000000000000),
>>
>> (0.3333333333333334, 0.6666666666666667, 0.5000000000000000),
>>
>> (0.6666666666666667, 0.3333333333333334, 0.5000000000000000)],
>>
>> pbc=(1, 1, 1),
>>
>> cell=[(4.260844986619441, -2.460000000000003, 0.000000000000000),
>>
>> (-0.000000000000000, 4.920000000000004, 0.000000000000000),
>>
>> (0.000000000000000, 0.000000000000000, 10.000000000000000)]
>>
>> )
>>
>> calc = GPAW(mode=PW(100),
>>
>> nbands=60,
>>
>> basis='dzp',
>>
>> eigensolver='rmm-diis',
>>
>> xc='LDA',
>>
>> kpts=(5, 5, 1),
>>
>> txt='graphene_1.txt')
>>
>> graphene.set_calculator(calc)
>>
>> graphene.get_potential_energy()
>>
>> calc.set(kpts=(20, 20, 1), fixdensity=True)
>>
>> calc.set(fixdensity=True)
>>
>> graphene.get_potential_energy()
>>
>> calc.diagonalize_full_hamiltonian(nbands=60)
>>
>> calc.write('graphene.gpw', 'all')
>>
>> df = DielectricFunction(calc='graphene.gpw',
>>
>> domega0=0.01,
>>
>> eta=0.3,
>>
>> ecut=20,
>>
>> txt='G_to_G.txt')
>>
>> q_c = [0.0, 0.0, 0.0]
>>
>> df.get_eels_spectrum(q_c=q_c, filename='G_to_G_EELS.csv')
>>
>> *******************************************************************************************************
>>
>> When I run this on 32 cores, I get these assertion errors:
>>
>> ? AssertionError
>>
>> calc.diagonalize_full_hamiltonian(nbands=60)
>>
>> "assert bd.comm.size == nprow * npcol" on line 746 of
>> $HOME/gpaw/0.11.0.12151/lib/python2.7/site-packages/gpaw/wavefunctions/pw.py
>>
>> ? GPAW CLEANUP (node 23): <type 'exceptions.AssertionError'>
>> occurred. Calling MPI_Abort!
>>
>> File
>> "/home/ufaserv1_g/pretha/soft/gpaw/0.11.0.12151/lib/python2.7/site-packages/gpaw/paw.py",
>> line 1000, in diagonalize_full_hamiltonian
>>
>> nbands, scalapack)
>>
>> What exactly are the quantities "bd.comm.size", "nprow" and "npcol",
>> and how can I avoid encountering these errors? I'm using the ARC2
>> machine at Leeds University:
>> https://hec.wiki.leeds.ac.uk/bin/view/Documentation/WebHome [1]
>>
>> Many thanks in advance for any tips.
>>
>> Trevor
>>
>> ************************
>>
>> Dr. Trevor P. Hardcastle
>>
>> EPSRC Doctoral Prize Fellow
>>
>> Room B20.C
>>
>> Engineering Building
>>
>> Institute for Materials Research
>>
>> University of Leeds
>>
>> LS2 9JT, United Kingdom
>>
>> t.p.hardcastle at leeds.ac.uk
>>
>> ************************
>>
>>
>>
>> Links:
>> ------
>> [1] https://hec.wiki.leeds.ac.uk/bin/view/Documentation/WebHome
>>
>> _______________________________________________
>> gpaw-users mailing list
>> gpaw-users at listserv.fysik.dtu.dk
>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>
>
> ------------------------------
>
> _______________________________________________
> gpaw-users mailing list
> gpaw-users at listserv.fysik.dtu.dk
> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>
> End of gpaw-users Digest, Vol 58, Issue 15
> ******************************************
More information about the gpaw-users
mailing list