[gpaw-users] gpaw-users Digest, Vol 58, Issue 15
Ask Hjorth Larsen
asklarsen at gmail.com
Tue Nov 18 11:26:11 CET 2014
Using scalapack on 8 cores may yield a speed-up of almost 8 of the
diagonalization. So by all means use scalapack whenever you do something
big-ish, no matter the number of cpus.
Also, you can ask for a scalapack distribution of (world.size // 8, 8, 64)
or something to do with len(calc.wfs.kpt_u) so you always get a compatible
distribution. This should remove the aforementioned hassle.
Best regards
Ask
El 18/11/2014 10:40, "Toma Susi" <toma.susi at univie.ac.at> escribió:
> I'm no expert on this, but I'd say you are better off testing
> parallelization with the number of cores (and k-points) you want to use.
>
> ------------------------------
> Dr. Toma Susi
> FWF Lise Meitner fellow
> University of Vienna, Austria
>
> http://mostlyphysics.wordpress.com
>
> *"Love is the only emotion that enhances our intelligence." *
> * -Humberto Maturana*
>
> On 18 Nov 2014, at 10:24, Trevor Hardcastle <T.P.Hardcastle at leeds.ac.uk>
> wrote:
>
> Hi Toma,
>
> I was using only 16/32 cores for testing purposes. I expect to be using up
> to ~512 cores for my larger systems once I've got to grips with effective
> parallelisation, so scalapack will certainly be a priority for me. Perhaps
> I'm better off testing with larger numbers of cores?
>
> Thanks,
>
> Trevor
>
> -----Original Message-----
> From: Toma Susi [mailto:toma.susi at univie.ac.at <toma.susi at univie.ac.at>]
> Sent: 18 November 2014 07:34
> To: Trevor Hardcastle
> Cc: gpaw-users at listserv.fysik.dtu.dk
> Subject: Re: gpaw-users Digest, Vol 58, Issue 15
>
> Hi Trevor,
>
> I would just run without Scalapack; with so few processors it is not
> necessary but makes parallelization a pain in the neck (+ weird errors).
>
> Cheers,
> Toma
>
>
>
> On 17.11.2014, at 20.34, gpaw-users-request at listserv.fysik.dtu.dk wrote:
>
>
> Send gpaw-users mailing list submissions to
>
> gpaw-users at listserv.fysik.dtu.dk
>
>
> To subscribe or unsubscribe via the World Wide Web, visit
>
> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>
> or, via email, send a message with subject or body 'help' to
>
> gpaw-users-request at listserv.fysik.dtu.dk
>
>
> You can reach the person managing the list at
>
> gpaw-users-owner at listserv.fysik.dtu.dk
>
>
> When replying, please edit your Subject line so it is more specific
>
> than "Re: Contents of gpaw-users digest..."
>
>
>
> Today's Topics:
>
>
> 1. Remember IRC-meeing tomorrow (Jens J?rgen Mortensen)
>
> 2. diagonalize_full_hamiltonian: Assertion error assert
>
> bd.comm.size == nprow * npcol (Trevor Hardcastle)
>
> 3. Re: diagonalize_full_hamiltonian: Assertion error assert
>
> bd.comm.size == nprow * npcol (Thomas Stenb?k Jauho)
>
>
>
> ----------------------------------------------------------------------
>
>
> Message: 1
>
> Date: Mon, 17 Nov 2014 16:19:13 +0100
>
> From: Jens J?rgen Mortensen <jensj at fysik.dtu.dk>
>
> To: gpaw-users <gpaw-users at listserv.fysik.dtu.dk>
>
> Subject: [gpaw-users] Remember IRC-meeing tomorrow
>
> Message-ID: <546A11F1.2060700 at fysik.dtu.dk>
>
> Content-Type: text/plain; charset="utf-8"; format=flowed
>
>
> 15:00 danish time: #gpaw on frnode.net
>
>
> Jens J?rgen
>
>
>
>
>
> ------------------------------
>
>
> Message: 2
>
> Date: Mon, 17 Nov 2014 19:16:49 +0000
>
> From: Trevor Hardcastle <T.P.Hardcastle at leeds.ac.uk>
>
> To: "gpaw-users at listserv.fysik.dtu.dk"
>
> <gpaw-users at listserv.fysik.dtu.dk>
>
> Subject: [gpaw-users] diagonalize_full_hamiltonian: Assertion error
>
> assert bd.comm.size == nprow * npcol
>
> Message-ID:
>
>
> <0FF15BFDE4634440B5F0D983CDF44C4E019B08E15C56 at HERMES8.ds.leeds.ac.uk>
>
> Content-Type: text/plain; charset="us-ascii"
>
>
> Hi all,
>
>
> Brand new GPAW! user here. I'm trying to get a basic EEL spectrum for
> N-doped graphene using gpaw/0.11.0.12151. The calculation works fine when
> using 16 cores, but fails when I try with 32 cores. The exact same .py
> script was used in both cases:
>
>
>
> **********************************************************************
>
> *********************************
>
> from __future__ import print_function
>
> import numpy as np
>
> from ase import Atoms
>
> from ase.parallel import paropen
>
> from gpaw import GPAW, FermiDirac, PW
>
> from gpaw.response.df import DielectricFunction
>
>
> graphene = Atoms('C7N',
>
>
> scaled_positions = [(0.8333333333333334,
> 0.1666666666666666, 0.5000000000000000),
>
> (0.8333333333333334,
> 0.6666666666666669, 0.5000000000000000),
>
> (0.3333333333333332,
> 0.1666666666666666, 0.5000000000000000),
>
> (0.1666666666666663,
> 0.3333333333333329, 0.5000000000000000),
>
> (0.6666666666666672,
> 0.8333333333333337, 0.5000000000000000),
>
> (0.1666666666666664,
> 0.8333333333333339, 0.5000000000000000),
>
> (0.3333333333333334,
> 0.6666666666666667, 0.5000000000000000),
>
> (0.6666666666666667,
> 0.3333333333333334, 0.5000000000000000)],
>
>
> pbc=(1, 1, 1),
>
>
> cell=[(4.260844986619441, -2.460000000000003,
> 0.000000000000000),
>
> (-0.000000000000000, 4.920000000000004,
> 0.000000000000000),
>
> (0.000000000000000, 0.000000000000000,
> 10.000000000000000)]
>
> )
>
>
>
> calc = GPAW(mode=PW(100),
>
> nbands=60,
>
> basis='dzp',
>
> eigensolver='rmm-diis',
>
> xc='LDA',
>
> kpts=(5, 5, 1),
>
> txt='graphene_1.txt')
>
>
> graphene.set_calculator(calc)
>
> graphene.get_potential_energy()
>
>
> calc.set(kpts=(20, 20, 1), fixdensity=True)
>
> calc.set(fixdensity=True)
>
> graphene.get_potential_energy()
>
>
> calc.diagonalize_full_hamiltonian(nbands=60)
>
> calc.write('graphene.gpw', 'all')
>
>
> df = DielectricFunction(calc='graphene.gpw',
>
> domega0=0.01,
>
> eta=0.3,
>
> ecut=20,
>
> txt='G_to_G.txt')
>
>
> q_c = [0.0, 0.0, 0.0]
>
>
> df.get_eels_spectrum(q_c=q_c, filename='G_to_G_EELS.csv')
>
> **********************************************************************
>
> *********************************
>
>
> When I run this on 32 cores, I get these assertion errors:
>
>
>
> * AssertionError
>
> calc.diagonalize_full_hamiltonian(nbands=60)
>
> "assert bd.comm.size == nprow * npcol" on line 746 of
>
> $HOME/gpaw/0.11.0.12151/lib/python2.7/site-packages/gpaw/wavefunctions
>
> /pw.py
>
>
>
> * GPAW CLEANUP (node 23): <type 'exceptions.AssertionError'>
> occurred. Calling MPI_Abort!
>
> File
> "/home/ufaserv1_g/pretha/soft/gpaw/0.11.0.12151/lib/python2.7/site-packages/gpaw/paw.py",
> line 1000, in diagonalize_full_hamiltonian
>
> nbands, scalapack)
>
>
>
> What exactly are the quantities "bd.comm.size", "nprow" and "npcol",
>
> and how can I avoid encountering these errors? I'm using the ARC2
>
> machine at Leeds University:
>
> https://hec.wiki.leeds.ac.uk/bin/view/Documentation/WebHome
>
>
> Many thanks in advance for any tips.
>
>
> Trevor
>
>
> ************************
>
> Dr. Trevor P. Hardcastle
>
> EPSRC Doctoral Prize Fellow
>
> Room B20.C
>
> Engineering Building
>
> Institute for Materials Research
>
> University of Leeds
>
> LS2 9JT, United Kingdom
>
> t.p.hardcastle at leeds.ac.uk<mailto:t.p.hardcastle at leeds.ac.uk
> <t.p.hardcastle at leeds.ac.uk>>
>
> ************************
>
>
> -------------- next part -------------- An HTML attachment was
>
> scrubbed...
>
> URL:
>
> <http://listserv.fysik.dtu.dk/pipermail/gpaw-users/attachments/2014111
>
> 7/ab7f2ba5/attachment-0001.html>
>
>
> ------------------------------
>
>
> Message: 3
>
> Date: Mon, 17 Nov 2014 11:33:58 -0800
>
> From: Thomas Stenb?k Jauho <s093006 at student.dtu.dk>
>
> To: gpaw-users at listserv.fysik.dtu.dk
>
> Subject: Re: [gpaw-users] diagonalize_full_hamiltonian: Assertion
>
> error assert bd.comm.size == nprow * npcol
>
> Message-ID: <d7cdd43ee4334a60cb9aa0816d358aa6 at student.dtu.dk>
>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
>
> Hi Trevor,
>
>
> To me it seems like an issue with scalapack and the parallelization
>
> you are trying.
>
>
> As I understand the code will try to parallelize over bands when using
>
> 32 cores (as it cannot parallelize 5x5x1 = 25 kpts on 32 cores).
>
>
> Therefore it will switch to band parallelization, however in order to
>
> do this successfully you should set scalapack correspondingly.
>
>
> As an example I use the following code for parallelizing the
>
> diagonalization over 8 or 16 cores respectively:
>
>
> calc.diagonalize_full_hamiltonian(nbands=nbands, scalapack=(4,4,32))
>
>
> This means nprow=4 and npcol=4, their product should equal the number
>
> of cores you're using, unless spin parallelization is included, in
>
> which case the product needs to be half the number of cores.
>
>
> Best,
>
> Thomas
>
>
>
>
>
> On 2014-11-17 11:16, Trevor Hardcastle wrote:
>
> Hi all,
>
>
> Brand new GPAW! user here. I'm trying to get a basic EEL spectrum for
>
> N-doped graphene using gpaw/0.11.0.12151. The calculation works fine
>
> when using 16 cores, but fails when I try with 32 cores. The exact
>
> same .py script was used in both cases:
>
>
> *********************************************************************
>
> **********************************
>
>
> from __future__ import print_function
>
>
> import numpy as np
>
>
> from ase import Atoms
>
>
> from ase.parallel import paropen
>
>
> from gpaw import GPAW, FermiDirac, PW
>
>
> from gpaw.response.df import DielectricFunction
>
>
> graphene = Atoms('C7N',
>
>
> scaled_positions = [(0.8333333333333334, 0.1666666666666666,
>
> 0.5000000000000000),
>
>
> (0.8333333333333334, 0.6666666666666669, 0.5000000000000000),
>
>
> (0.3333333333333332, 0.1666666666666666, 0.5000000000000000),
>
>
> (0.1666666666666663, 0.3333333333333329, 0.5000000000000000),
>
>
> (0.6666666666666672, 0.8333333333333337, 0.5000000000000000),
>
>
> (0.1666666666666664, 0.8333333333333339, 0.5000000000000000),
>
>
> (0.3333333333333334, 0.6666666666666667, 0.5000000000000000),
>
>
> (0.6666666666666667, 0.3333333333333334, 0.5000000000000000)],
>
>
> pbc=(1, 1, 1),
>
>
> cell=[(4.260844986619441, -2.460000000000003, 0.000000000000000),
>
>
> (-0.000000000000000, 4.920000000000004, 0.000000000000000),
>
>
> (0.000000000000000, 0.000000000000000, 10.000000000000000)]
>
>
> )
>
>
> calc = GPAW(mode=PW(100),
>
>
> nbands=60,
>
>
> basis='dzp',
>
>
> eigensolver='rmm-diis',
>
>
> xc='LDA',
>
>
> kpts=(5, 5, 1),
>
>
> txt='graphene_1.txt')
>
>
> graphene.set_calculator(calc)
>
>
> graphene.get_potential_energy()
>
>
> calc.set(kpts=(20, 20, 1), fixdensity=True)
>
>
> calc.set(fixdensity=True)
>
>
> graphene.get_potential_energy()
>
>
> calc.diagonalize_full_hamiltonian(nbands=60)
>
>
> calc.write('graphene.gpw', 'all')
>
>
> df = DielectricFunction(calc='graphene.gpw',
>
>
> domega0=0.01,
>
>
> eta=0.3,
>
>
> ecut=20,
>
>
> txt='G_to_G.txt')
>
>
> q_c = [0.0, 0.0, 0.0]
>
>
> df.get_eels_spectrum(q_c=q_c, filename='G_to_G_EELS.csv')
>
>
> *********************************************************************
>
> **********************************
>
>
> When I run this on 32 cores, I get these assertion errors:
>
>
> ? AssertionError
>
>
> calc.diagonalize_full_hamiltonian(nbands=60)
>
>
> "assert bd.comm.size == nprow * npcol" on line 746 of
>
> $HOME/gpaw/0.11.0.12151/lib/python2.7/site-packages/gpaw/wavefunction
>
> s/pw.py
>
>
> ? GPAW CLEANUP (node 23): <type 'exceptions.AssertionError'>
>
> occurred. Calling MPI_Abort!
>
>
> File
>
> "/home/ufaserv1_g/pretha/soft/gpaw/0.11.0.12151/lib/python2.7/site-pa
>
> ckages/gpaw/paw.py", line 1000, in diagonalize_full_hamiltonian
>
>
> nbands, scalapack)
>
>
> What exactly are the quantities "bd.comm.size", "nprow" and "npcol",
>
> and how can I avoid encountering these errors? I'm using the ARC2
>
> machine at Leeds University:
>
> https://hec.wiki.leeds.ac.uk/bin/view/Documentation/WebHome [1]
>
>
> Many thanks in advance for any tips.
>
>
> Trevor
>
>
> ************************
>
>
> Dr. Trevor P. Hardcastle
>
>
> EPSRC Doctoral Prize Fellow
>
>
> Room B20.C
>
>
> Engineering Building
>
>
> Institute for Materials Research
>
>
> University of Leeds
>
>
> LS2 9JT, United Kingdom
>
>
> t.p.hardcastle at leeds.ac.uk
>
>
> ************************
>
>
>
>
> Links:
>
> ------
>
> [1] https://hec.wiki.leeds.ac.uk/bin/view/Documentation/WebHome
>
>
> _______________________________________________
>
> gpaw-users mailing list
>
> gpaw-users at listserv.fysik.dtu.dk
>
> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>
>
>
> ------------------------------
>
>
> _______________________________________________
>
> gpaw-users mailing list
>
> gpaw-users at listserv.fysik.dtu.dk
>
> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>
>
> End of gpaw-users Digest, Vol 58, Issue 15
>
> ******************************************
>
>
> _______________________________________________
> gpaw-users mailing list
> gpaw-users at listserv.fysik.dtu.dk
> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.fysik.dtu.dk/pipermail/gpaw-users/attachments/20141118/85831597/attachment-0001.html>
More information about the gpaw-users
mailing list