[gpaw-users] Query about parallelism scheme in GPAW

[Jens Jørgen Mortensen]

On 1/25/19 10:42 AM, Richard Terrett wrote:
>
> Hi,
>
>
> exporting OMP_NUM_THREADS=1 solved the problem, and I can confirm a 
> decent parallel speedup.
>
>
> Thanks for your assistance!
>

No problem.


Jens Jørgen

(CC to gpaw-users mail list)


>
> Dr Richard Terrett
>
> Postdoctoral Fellow, Computational Quantum Chemistry Group
>
> Research School of Chemistry, ANU
>
>
>
> ------------------------------------------------------------------------
> *From:* Jens Jørgen Mortensen <jjmo at dtu.dk>
> *Sent:* 25 January 2019 16:51
> *To:* Richard Terrett; gpaw-users at listserv.fysik.dtu.dk
> *Subject:* Re: [gpaw-users] Query about parallelism scheme in GPAW
> On 1/25/19 5:26 AM, Richard Terrett via gpaw-users wrote:
>>
>> Hi,
>>
>>
>> I'm trying to probe efficient parallelism of GPAW on a desktop system 
>> (4 cores, 8 threads w/ hyperthreading). I note that when I run mpirun 
>> -n x gpaw-python where x > 1, I get 8*x gpaw-python processes. Where 
>> x = 8, that's 64 gpaw-python processes, which far exceeds the number 
>> of logical cores on my system. I note that for the system that I'm 
>> looking at (PW, 800 eV ecut, 64 kpts, PBE, 13 atoms), I get slower 
>> SCF iterations for -n 8 than for -n 1, indicating that contention is 
>> occurring. Nevertheless, when I run with -n 1, GPAW recognises that I 
>> am only running on '1 CPU' and therefore as far as I can tell does 
>> not parallelise over kpoints, bands, etc. Moreover, I cannot supply 
>> values greater than 1 to the relevant parallel dict keys in this 
>> scenario without GPAW crashing.
>>
>>
>> To be clear, when I run -n 1, I do get 8 GPAW processes, although the 
>> system does not get saturated.
>>
>>
>> Is this expected GPAW behaviour? I can't really tell if GPAW is 
>> effectively parallelising or just spinning its wheels, and I don't 
>> know how to nerf GPAW to run fewer than 8 threads for comparison. 
>> Even if I just run run my script via python3 (rather than 
>> gpaw-python) it still spawns 16 threads during the SCF loop, running 
>> at about the same speed as a job initiated with mpirun.
>>
>>
>> My specs: gpaw-1.5.1, ase-3.18.0b1 (reported by python import) or 
>> ase-3.15.0 (installed from ubuntu repos), mpirun (OpenMPI) 2.1.1, 
>> Intel Core i7-4790 (4 cores 8 threads), Ubuntu 18.04 x86_64.
>>
>>
>> Oh, I'm also using the Davidson eigensolver, but rmm-diis did not 
>> seem to affect the speed meaningfully.
>>
>>
>> Any help that can be provided regarding this question is appreciated.
>>
>
> I guess it is your BLAS library (that NumPy and GPAW link to) that is 
> starting those threads.  You can turn that off by setting 
> OMP_NUM_THREADS=1.
>
>
> Jens Jørgen
>
>
>>
>> Dr Richard Terrett
>>
>> Postdoctoral Fellow, Computational Quantum Chemistry Group
>>
>> Research School of Chemistry, ANU
>>
>>
>> _______________________________________________
>> gpaw-users mailing list
>> gpaw-users at listserv.fysik.dtu.dk  <mailto:gpaw-users at listserv.fysik.dtu.dk>
>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.fysik.dtu.dk/pipermail/gpaw-users/attachments/20190125/03db51dc/attachment-0001.html>