[gpaw-users] GPAW and InfiniBand

[Marcin Dulak]

Hi,

On 11/23/12 11:29, Morten Bjørn Bakkedal wrote:
> I'm using GPAW on DTU's central Xeon based HPC cluster which is interconnected
> with InfiniBand. I'm currently doing computations on pretty large systems with
> 64 atoms and 576 bands on 64 computer cores.
>
> Now I thinking about how to be sure that the processes are communicating using
> the fast InfiniBand connection. Do I need to add any compiler flags to the
> customize.py to compile GPAW to use the fast connection? As I wrote a few days
> ago, I'm using the ACML library.
you need to use the openmpi compiled with infiniband support.
Hopefully the cluster provides one.
A simple test will be to run excluding ethernet:
mpiexec --mca btl ^tcp gpaw-python script.py
or:
mpiexec --mca btl ^tcp --verbose --mca btl_openib_verbose 1 ...

You system does not seem very large, do you really need 64 cores?
Usually this is limited by the available memory per core, so run gpaw 
first with --dry-run=N
(see 
https://wiki.fysik.dtu.dk/gpaw/documentation/parallel_runs/parallel_runs.html)
and make sure that the reported estimated memory per core (labelled as 
Calculator)
is 50-75% of the available memory per core as gpaw quite often 
underestimates the required memory.

Best regards,

Marcin
>


-- 
***********************************
  
Marcin Dulak
Technical University of Denmark
Department of Physics
Building 307, Room 229
DK-2800 Kongens Lyngby
Denmark
Tel.: (+45) 4525 3157
Fax.: (+45) 4593 2399
email: Marcin.Dulak at fysik.dtu.dk

***********************************