[gpaw-users] diagonalize_full_hamiltonian: Assertion error assert bd.comm.size == nprow * npcol

Trevor Hardcastle T.P.Hardcastle at leeds.ac.uk
Tue Nov 18 19:14:29 CET 2014 

That's interesting to hear actually; I've been quite successful in producing EEL spectra with in-plane q-dependence along the G-M and G-K directions for a simple 2-atom graphene unit cell. The plots I obtained look quite similar, certainly in general character, to Fig. 2a from Yan et al. PRL 106, 146803 (2011).  (That does actually bring me to another question, but I'll ask it in a separate thread.)

As for my original problem, I'm having more success now by following Jens' suggestion of splitting the calculation into three components and allocating core numbers appropriately. Using parallel={'band': 1} seems to have worked nicely for the diagonalize_full_hamiltonian method; I was able to run the calculation quite quickly using 72 cores with a 12x12x1 kpoints grid (72 kpts in the IBZ).

Trevor

-----Original Message-----
From: gpaw-users-bounces at listserv.fysik.dtu.dk [mailto:gpaw-users-bounces at listserv.fysik.dtu.dk] On Behalf Of Ask Hjorth Larsen
Sent: 18 November 2014 16:20
To: Marcin Dulak
Cc: gpaw-users
Subject: Re: [gpaw-users] diagonalize_full_hamiltonian: Assertion error assert bd.comm.size == nprow * npcol

Apparently some bad numbers are passed sometimes when the system is too small.  We never solved this because it has never affected a system of significant size and nobody likes or has time to mess with that stuff when it isn't breaking something they need.  Hmmm.  This doesn't answer any questions, but someone could look into the error and try to fix it (not me though, sorry).

Best regards
Ask

2014-11-18 13:15 GMT+01:00 Marcin Dulak <Marcin.Dulak at fysik.dtu.dk>:
> On 11/18/2014 10:46 AM, Jens Jørgen Mortensen wrote:
>>
>> On 11/17/2014 08:33 PM, Thomas Stenbæk Jauho wrote:
>>>
>>> Hi Trevor,
>>>
>>> To me it seems like an issue with scalapack and the parallelization 
>>> you are trying.
>>>
>>> As I understand the code will try to parallelize over bands when 
>>> using 32 cores (as it cannot parallelize 5x5x1 = 25 kpts on 32 cores).
>>>
>>> Therefore it will switch to band parallelization, however in order 
>>> to do this successfully you should set scalapack correspondingly.
>>>
>>> As an example I use the following code for parallelizing the 
>>> diagonalization over 8 or 16 cores respectively:
>>>
>>> calc.diagonalize_full_hamiltonian(nbands=nbands, scalapack=(4,4,32))
>>>
>>> This means nprow=4 and npcol=4, their product should equal the 
>>> number of cores you're using, unless spin parallelization is 
>>> included, in which case the product needs to be half the number of cores.
>>
>>
>> With recent versions of GPAW, one can just use "scalapack=True" and 
>> it will figure out what ScaLapack parameters to use.  With the latest 
>> development version (from today), you don't have to set "scalapack" 
>> at all - it will use ScaLapack if there is parallelization over bands.
>
> is this good? - we had problems in the past where scalapack was giving 
> wrong total energies:
> https://listserv.fysik.dtu.dk/pipermail/gpaw-users/2013-January/001972
> .html
>
> Best regards,
>
> Marcin
>
>>
>> I normally split such calculations in three parts: 1) ground-state, 
>> 2) full diagonalization, and then finally 3) calculation of the 
>> response function.  Then I can choose freely the most optimal number 
>> of processes for each step.
>>
>> There is a helper function:
>>
>> https://wiki.fysik.dtu.dk/gpaw/epydoc/gpaw.fulldiag-module.html
>>
>> that will do step 2 if you have a gpw-file containing the 
>> ground-state density.
>>
>> If you don't want to use ScaLapack then you can force GPAW to 
>> parallelize over spins and k-points (possibly with a bit of load imbalance) like this:
>>
>>     calc = GPAW('gs.gpw', parallel={'band': 1})
>>     calc.diagonalize_full_hamiltonian(60)
>>     calc.write('gs-all.gpw', 'all')
>>
>> or equivalently:
>>
>>     fulldiag('gs.gpw', 60)
>>
>> Jens Jørgen
>>
>>
>>
>>
>>
>>
>>>
>>> Best,
>>> Thomas
>>>
>>>
>>>
>>>
>>> On 2014-11-17 11:16, Trevor Hardcastle wrote:
>>>>
>>>> Hi all,
>>>>
>>>> Brand new GPAW! user here. I'm trying to get a basic EEL spectrum 
>>>> for N-doped graphene using gpaw/0.11.0.12151. The calculation works 
>>>> fine when using 16 cores, but fails when I try with 32 cores. The 
>>>> exact same .py script was used in both cases:
>>>>
>>>>
>>>> *******************************************************************
>>>> ************************************
>>>>
>>>> from __future__ import print_function
>>>>
>>>> import numpy as np
>>>>
>>>> from ase import Atoms
>>>>
>>>> from ase.parallel import paropen
>>>>
>>>> from gpaw import GPAW, FermiDirac, PW
>>>>
>>>> from gpaw.response.df import DielectricFunction
>>>>
>>>> graphene = Atoms('C7N',
>>>>
>>>>  scaled_positions = [(0.8333333333333334, 0.1666666666666666, 
>>>> 0.5000000000000000),
>>>>
>>>>  (0.8333333333333334, 0.6666666666666669, 0.5000000000000000),
>>>>
>>>>  (0.3333333333333332, 0.1666666666666666, 0.5000000000000000),
>>>>
>>>>  (0.1666666666666663, 0.3333333333333329, 0.5000000000000000),
>>>>
>>>>  (0.6666666666666672, 0.8333333333333337, 0.5000000000000000),
>>>>
>>>>  (0.1666666666666664, 0.8333333333333339, 0.5000000000000000),
>>>>
>>>>  (0.3333333333333334, 0.6666666666666667, 0.5000000000000000),
>>>>
>>>>  (0.6666666666666667, 0.3333333333333334, 0.5000000000000000)],
>>>>
>>>>  pbc=(1, 1, 1),
>>>>
>>>>  cell=[(4.260844986619441, -2.460000000000003, 0.000000000000000),
>>>>
>>>>  (-0.000000000000000, 4.920000000000004, 0.000000000000000),
>>>>
>>>>  (0.000000000000000, 0.000000000000000, 10.000000000000000)]
>>>>
>>>>  )
>>>>
>>>> calc = GPAW(mode=PW(100),
>>>>
>>>>  nbands=60,
>>>>
>>>>  basis='dzp',
>>>>
>>>>  eigensolver='rmm-diis',
>>>>
>>>>  xc='LDA',
>>>>
>>>>  kpts=(5, 5, 1),
>>>>
>>>>  txt='graphene_1.txt')
>>>>
>>>> graphene.set_calculator(calc)
>>>>
>>>> graphene.get_potential_energy()
>>>>
>>>> calc.set(kpts=(20, 20, 1), fixdensity=True)
>>>>
>>>> calc.set(fixdensity=True)
>>>>
>>>> graphene.get_potential_energy()
>>>>
>>>> calc.diagonalize_full_hamiltonian(nbands=60)
>>>>
>>>> calc.write('graphene.gpw', 'all')
>>>>
>>>> df = DielectricFunction(calc='graphene.gpw',
>>>>
>>>>  domega0=0.01,
>>>>
>>>>  eta=0.3,
>>>>
>>>>  ecut=20,
>>>>
>>>>  txt='G_to_G.txt')
>>>>
>>>> q_c = [0.0, 0.0, 0.0]
>>>>
>>>> df.get_eels_spectrum(q_c=q_c, filename='G_to_G_EELS.csv')
>>>>
>>>>
>>>> *******************************************************************
>>>> ************************************
>>>>
>>>> When I run this on 32 cores, I get these assertion errors:
>>>>
>>>> · AssertionError
>>>>
>>>>  calc.diagonalize_full_hamiltonian(nbands=60)
>>>>
>>>> "assert bd.comm.size == nprow * npcol" on line 746 of
>>>>
>>>> $HOME/gpaw/0.11.0.12151/lib/python2.7/site-packages/gpaw/wavefuncti
>>>> ons/pw.py
>>>>
>>>> · GPAW CLEANUP (node 23): <type 'exceptions.AssertionError'> 
>>>> occurred. Calling MPI_Abort!
>>>>
>>>>  File
>>>>
>>>> "/home/ufaserv1_g/pretha/soft/gpaw/0.11.0.12151/lib/python2.7/site-
>>>> packages/gpaw/paw.py", line 1000, in diagonalize_full_hamiltonian
>>>>
>>>>  nbands, scalapack)
>>>>
>>>> What exactly are the quantities "bd.comm.size", "nprow" and 
>>>> "npcol", and how can I avoid encountering these errors? I'm using 
>>>> the ARC2 machine at Leeds University:
>>>> https://hec.wiki.leeds.ac.uk/bin/view/Documentation/WebHome [1]
>>>>
>>>> Many thanks in advance for any tips.
>>>>
>>>> Trevor
>>>>
>>>> ************************
>>>>
>>>> Dr. Trevor P. Hardcastle
>>>>
>>>> EPSRC Doctoral Prize Fellow
>>>>
>>>> Room B20.C
>>>>
>>>> Engineering Building
>>>>
>>>> Institute for Materials Research
>>>>
>>>> University of Leeds
>>>>
>>>> LS2 9JT, United Kingdom
>>>>
>>>> t.p.hardcastle at leeds.ac.uk
>>>>
>>>> ************************
>>>>
>>>>
>>>>
>>>> Links:
>>>> ------
>>>> [1] https://hec.wiki.leeds.ac.uk/bin/view/Documentation/WebHome
>>>>
>>>> _______________________________________________
>>>> gpaw-users mailing list
>>>> gpaw-users at listserv.fysik.dtu.dk
>>>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>>>
>>> _______________________________________________
>>> gpaw-users mailing list
>>> gpaw-users at listserv.fysik.dtu.dk
>>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>>
>>
>> _______________________________________________
>> gpaw-users mailing list
>> gpaw-users at listserv.fysik.dtu.dk
>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>
>
>
> _______________________________________________
> gpaw-users mailing list
> gpaw-users at listserv.fysik.dtu.dk
> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users

_______________________________________________
gpaw-users mailing list
gpaw-users at listserv.fysik.dtu.dk
https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users

More information about the gpaw-users mailing list