[gpaw-users] diagonalize_full_hamiltonian: Assertion error assert bd.comm.size == nprow * npcol

Tue Nov 18 13:15:22 CET 2014

On 11/18/2014 10:46 AM, Jens Jørgen Mortensen wrote:
> On 11/17/2014 08:33 PM, Thomas Stenbæk Jauho wrote:
>> Hi Trevor,
>>
>> To me it seems like an issue with scalapack and the parallelization 
>> you are trying.
>>
>> As I understand the code will try to parallelize over bands when 
>> using 32 cores (as it cannot parallelize 5x5x1 = 25 kpts on 32 cores).
>>
>> Therefore it will switch to band parallelization, however in order to 
>> do this successfully you should set scalapack correspondingly.
>>
>> As an example I use the following code for parallelizing the 
>> diagonalization over 8 or 16 cores respectively:
>>
>> calc.diagonalize_full_hamiltonian(nbands=nbands, scalapack=(4,4,32))
>>
>> This means nprow=4 and npcol=4, their product should equal the number 
>> of cores you're using, unless spin parallelization is included, in 
>> which case the product needs to be half the number of cores.
>
> With recent versions of GPAW, one can just use "scalapack=True" and it 
> will figure out what ScaLapack parameters to use.  With the latest 
> development version (from today), you don't have to set "scalapack" at 
> all - it will use ScaLapack if there is parallelization over bands.
is this good? - we had problems in the past where scalapack was giving 
wrong total energies:
https://listserv.fysik.dtu.dk/pipermail/gpaw-users/2013-January/001972.html

Best regards,

Marcin
>
> I normally split such calculations in three parts: 1) ground-state, 2) 
> full diagonalization, and then finally 3) calculation of the response 
> function.  Then I can choose freely the most optimal number of 
> processes for each step.
>
> There is a helper function:
>
> https://wiki.fysik.dtu.dk/gpaw/epydoc/gpaw.fulldiag-module.html
>
> that will do step 2 if you have a gpw-file containing the ground-state 
> density.
>
> If you don't want to use ScaLapack then you can force GPAW to 
> parallelize over spins and k-points (possibly with a bit of load 
> imbalance) like this:
>
>     calc = GPAW('gs.gpw', parallel={'band': 1})
>     calc.diagonalize_full_hamiltonian(60)
>     calc.write('gs-all.gpw', 'all')
>
> or equivalently:
>
>     fulldiag('gs.gpw', 60)
>
> Jens Jørgen
>
>
>
>
>
>
>>
>> Best,
>> Thomas
>>
>>
>>
>>
>> On 2014-11-17 11:16, Trevor Hardcastle wrote:
>>> Hi all,
>>>
>>> Brand new GPAW! user here. I'm trying to get a basic EEL spectrum for
>>> N-doped graphene using gpaw/0.11.0.12151. The calculation works fine
>>> when using 16 cores, but fails when I try with 32 cores. The exact
>>> same .py script was used in both cases:
>>>
>>> ******************************************************************************************************* 
>>>
>>>
>>> from __future__ import print_function
>>>
>>> import numpy as np
>>>
>>> from ase import Atoms
>>>
>>> from ase.parallel import paropen
>>>
>>> from gpaw import GPAW, FermiDirac, PW
>>>
>>> from gpaw.response.df import DielectricFunction
>>>
>>> graphene = Atoms('C7N',
>>>
>>>  scaled_positions = [(0.8333333333333334, 0.1666666666666666,
>>> 0.5000000000000000),
>>>
>>>  (0.8333333333333334, 0.6666666666666669, 0.5000000000000000),
>>>
>>>  (0.3333333333333332, 0.1666666666666666, 0.5000000000000000),
>>>
>>>  (0.1666666666666663, 0.3333333333333329, 0.5000000000000000),
>>>
>>>  (0.6666666666666672, 0.8333333333333337, 0.5000000000000000),
>>>
>>>  (0.1666666666666664, 0.8333333333333339, 0.5000000000000000),
>>>
>>>  (0.3333333333333334, 0.6666666666666667, 0.5000000000000000),
>>>
>>>  (0.6666666666666667, 0.3333333333333334, 0.5000000000000000)],
>>>
>>>  pbc=(1, 1, 1),
>>>
>>>  cell=[(4.260844986619441, -2.460000000000003, 0.000000000000000),
>>>
>>>  (-0.000000000000000, 4.920000000000004, 0.000000000000000),
>>>
>>>  (0.000000000000000, 0.000000000000000, 10.000000000000000)]
>>>
>>>  )
>>>
>>> calc = GPAW(mode=PW(100),
>>>
>>>  nbands=60,
>>>
>>>  basis='dzp',
>>>
>>>  eigensolver='rmm-diis',
>>>
>>>  xc='LDA',
>>>
>>>  kpts=(5, 5, 1),
>>>
>>>  txt='graphene_1.txt')
>>>
>>> graphene.set_calculator(calc)
>>>
>>> graphene.get_potential_energy()
>>>
>>> calc.set(kpts=(20, 20, 1), fixdensity=True)
>>>
>>> calc.set(fixdensity=True)
>>>
>>> graphene.get_potential_energy()
>>>
>>> calc.diagonalize_full_hamiltonian(nbands=60)
>>>
>>> calc.write('graphene.gpw', 'all')
>>>
>>> df = DielectricFunction(calc='graphene.gpw',
>>>
>>>  domega0=0.01,
>>>
>>>  eta=0.3,
>>>
>>>  ecut=20,
>>>
>>>  txt='G_to_G.txt')
>>>
>>> q_c = [0.0, 0.0, 0.0]
>>>
>>> df.get_eels_spectrum(q_c=q_c, filename='G_to_G_EELS.csv')
>>>
>>> ******************************************************************************************************* 
>>>
>>>
>>> When I run this on 32 cores, I get these assertion errors:
>>>
>>> · AssertionError
>>>
>>>  calc.diagonalize_full_hamiltonian(nbands=60)
>>>
>>> "assert bd.comm.size == nprow * npcol" on line 746 of
>>> $HOME/gpaw/0.11.0.12151/lib/python2.7/site-packages/gpaw/wavefunctions/pw.py 
>>>
>>>
>>> · GPAW CLEANUP (node 23): <type 'exceptions.AssertionError'>
>>> occurred. Calling MPI_Abort!
>>>
>>>  File
>>> "/home/ufaserv1_g/pretha/soft/gpaw/0.11.0.12151/lib/python2.7/site-packages/gpaw/paw.py", 
>>>
>>> line 1000, in diagonalize_full_hamiltonian
>>>
>>>  nbands, scalapack)
>>>
>>> What exactly are the quantities "bd.comm.size", "nprow" and "npcol",
>>> and how can I avoid encountering these errors? I'm using the ARC2
>>> machine at Leeds University:
>>> https://hec.wiki.leeds.ac.uk/bin/view/Documentation/WebHome [1]
>>>
>>> Many thanks in advance for any tips.
>>>
>>> Trevor
>>>
>>> ************************
>>>
>>> Dr. Trevor P. Hardcastle
>>>
>>> EPSRC Doctoral Prize Fellow
>>>
>>> Room B20.C
>>>
>>> Engineering Building
>>>
>>> Institute for Materials Research
>>>
>>> University of Leeds
>>>
>>> LS2 9JT, United Kingdom
>>>
>>> t.p.hardcastle at leeds.ac.uk
>>>
>>> ************************
>>>
>>>
>>>
>>> Links:
>>> ------
>>> [1] https://hec.wiki.leeds.ac.uk/bin/view/Documentation/WebHome
>>>
>>> _______________________________________________
>>> gpaw-users mailing list
>>> gpaw-users at listserv.fysik.dtu.dk
>>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>> _______________________________________________
>> gpaw-users mailing list
>> gpaw-users at listserv.fysik.dtu.dk
>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>
> _______________________________________________
> gpaw-users mailing list
> gpaw-users at listserv.fysik.dtu.dk
> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users