[gpaw-users] general comment on memory leaks.
abhishek khetan
askhetan at gmail.com
Wed Jan 20 11:19:23 CET 2016
I think I have figured out exactly where the problem lies, but not what is
causing it.
First, just to give you what our two clusters here are like (in case they
may be of help):
Cluster1:
Chassis: 14x Dell PowerEdge C6100 *(Means 14 nodes on this chassis/cluster)*
Processor/Node: 2x Intel Xeon X5670 (6-core) *(Means a total of 2x6=12
processors per node)*
Memory/Node: 48 GByte (12x 4 GByte, 1333MHz) *(Means atleast 3.5 Gbs of
actual resident memory available per core)*
Interconnect:Infiniband QDR Dual Port 40Gb/s (non-blocking)
<http://gottfried.itv.rwth-aachen.de/wiki/lib/exe/fetch.php?media=computing:is5030_35.pdf>
File System: lustre file system
Operating System: Scientific Linux 6.4
Cluster2:
Blades: 6x Dell PowerEdge M620 *(Means 6 nodes per chassis/cluster)*
Processor/Blade: 2x Intel Xeon E5-2660v2 (10-core) *(Means a total of
2x10=20 processors per node)*
Memory/Blade: 256 GByte *(Means atleast 12.5 Gbs of actual resident memory
available per core)*
Interconnect: Infiniband FDR-10
<http://www.mellanox.com/related-docs/whitepapers/WP_FDR_InfiniBand_is_Here.pdf>
File System: lustre file system
Operating System: Scientific Linux 6.4
I did an experiment I some low memory jobs (with kpts=1x1x1) on 12 and 24
processors on Cluster1 and I also ran some higher memory jobs (with
kpts=1x1x2) on 20 and 40 processors on Cluster2.
In both cases, when the jobs did not span over more than one node, which
means 12 procs on Cluster1 for low mem jobs and 20 procs on Cluster2 for
high mem jobs, they ran perfectly well EVERY SINGLE TIME.
However, as I increased the number of processors from 12 (1 node) to 24 (2
nodes) for the low mem jobs on Cluster1, and 20 (1 node) to 40 (2 nodes)
for the higher mem jobs on Cluster2, the behaviour is totally erratic.
Sometimes they start, other times they give the same segfault error, which
I have described previously in this post. Another interesting feature was
that the more number of processors (and therefore nodes) i run the jobs on,
the more difficult it is to get to jobs to start. In all simplicity, the
number of times the jobs crashed was found to be an exponentially
increasing function of the number of nodes involved. As pseudo-scientific
this sounds, its actually what is happening. I have no clue why.
Although, this clearly indicates a problem with the inter-node
communication here on the cluster, because on single nodes, there is no
problem at all. I have provided you with the exact technical details so
that maybe you can let me know if its a known problem on Infiniband FDR or
QDR interconnections. Could there be a problem in my compilation? Seems to
me not because even on 3 or 4 nodes, the jobs do start sometimes, if I am
lucky.
Any help is greatly appreciated.
On Mon, Jan 18, 2016 at 7:35 PM, abhishek khetan <askhetan at gmail.com> wrote:
> You're right, the word memory leak is a wrong description. I made the
> mistake of invariable associating it with the seg fault error, which is
> what it actually is. I will make these tests and get back.
>
>
> On Mon, Jan 18, 2016 at 6:32 PM, Ask Hjorth Larsen <asklarsen at gmail.com>
> wrote:
>
>> Why are you so sure that there are memory leaks? So far we have only
>> seen indications that a lot of memory is allocated.
>>
>> You could for example lower the grid spacing until it runs, then check
>> if memory usage increases linearly with subsequent identical
>> calculations. That would indicate a memory leak. If you do not
>> observe this behaviour, then I don't know what you are seeing, but it
>> is certainly not a memory leak!
>>
>> 2016-01-18 13:26 GMT+01:00 abhishek khetan <askhetan at gmail.com>:
>> > I tried using the cluster interactively, and it gives me the output as
>> > below. I couldn't make the r_memusage function work but its easily
>> visible
>> > that the memory requirements are quite modest. I do not know why there
>> is
>> > seg fault when I allocate it in the regular cluster for production jobs.
>> >
>> > ___ ___ ___ _ _ _
>> > | | |_ | | | |
>> > | | | | | . | | | |
>> > |__ | _|___|_____| 0.12.0.13279
>> > |___|_|
>> >
>> > User: ak498084 at linuxbmc0002.rz.RWTH-Aachen.DE
>> > Date: Mon Jan 18 13:22:24 2016
>> > Arch: x86_64
>> > Pid: 20443
>> > Python: 2.7.9
>> > gpaw: /home/ak498084/Utility/GPAW/gpaw_devel/gpaw-0.12/gpaw
>> > _gpaw:
>> >
>> /home/ak498084/Utility/GPAW/gpaw_devel/gpaw-0.12/build/bin.linux-x86_64-2.7/gpaw-python
>> > ase: /home/ak498084/Utility/GPAW/gpaw_devel/ase/ase (version 3.10.0)
>> > numpy:
>> > /usr/local_rwth/sw/python/2.7.9/x86_64/lib/python2.7/site-packages/numpy
>> > (version 1.9.1)
>> > scipy:
>> > /usr/local_rwth/sw/python/2.7.9/x86_64/lib/python2.7/site-packages/scipy
>> > (version 0.15.1)
>> > units: Angstrom and eV
>> > cores: 32
>> >
>> > Memory estimate
>> > ---------------
>> > Process memory now: 75.02 MiB
>> > Calculator 1145.24 MiB
>> > Density 56.04 MiB
>> > Arrays 15.91 MiB
>> > Localized functions 35.58 MiB
>> > Mixer 4.55 MiB
>> > Hamiltonian 23.19 MiB
>> > Arrays 11.82 MiB
>> > XC 0.00 MiB
>> > Poisson 8.81 MiB
>> > vbar 2.56 MiB
>> > Wavefunctions 1066.01 MiB
>> > Arrays psit_nG 523.69 MiB
>> > Eigensolver 2.29 MiB
>> > Projections 2.06 MiB
>> > Projectors 4.17 MiB
>> > Overlap op 533.81 MiB
>> >
>> >
>> > On Mon, Jan 18, 2016 at 1:01 PM, abhishek khetan <askhetan at gmail.com>
>> wrote:
>> >>
>> >> Dear Marcin, and Ask,
>> >>
>> >> I am indeed on this cluster. And I have already used both these tools.
>> >> When I use the r_memusage (to check the peak physical memory), the peak
>> >> physical memory is in the order of a few MBs and the process gets
>> killed
>> >> right as the beginning with the output only as:
>> >>
>> >> | | |_ | | | |
>> >> | | | | | . | | | |
>> >> |__ | _|___|_____| 0.12.0.13279
>> >> |___|_|
>> >>
>> >>
>> >> The same is not the case when I take a pre-converged systems and run
>> the
>> >> r_memusage script. It shows me a good 2.5 GBs (and rising) before I
>> kill the
>> >> process, as I can see its running fine. This is what I mean by saying
>> that
>> >> the allocation doesn't even start for these unconverged cases. Using
>> >> eigensolver=RMM_DIIS(keep_htpsit=False), has the exact same problems.
>> Is
>> >> there a way, I can trick gpaw into giving the cluster much less of a
>> >> requirement. I want to try this because, as I have mentioned, at the
>> peak
>> >> condition my jobs don't need more than 2 GB per core and I'm providing
>> it 8
>> >> GB usually (albiet, to no use).
>> >>
>> >> Best,
>> >>
>> >>
>> >> On Sat, Jan 16, 2016 at 1:10 PM, Marcin Dulak <mdul at dtu.dk> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> are you one this cluster?
>> >>> https://doc.itc.rwth-aachen.de/display/CC/r_memusage
>> >>>
>> >>>
>> https://doc.itc.rwth-aachen.de/display/CC/Resource+limitations+on+dialog+systems
>> >>> It may be that the batch system (LSF) kills your jobs that exceed
>> given
>> >>> resident memory.
>> >>> The two links above may help you to diagnose that.
>> >>> I recall GPAW's memory estimate is not very accurate for standard
>> >>> ground-state, PW or GRID mode jobs
>> >>> (~20%) and may be very inaccurate (order of magnitude) for VDW or LCAO
>> >>> jobs (Ask correct me if this is not the case anymore).
>> >>>
>> >>> Best regards,
>> >>>
>> >>> Marcin
>> >>> _______________________________________________
>> >>> gpaw-users mailing list
>> >>> gpaw-users at listserv.fysik.dtu.dk
>> >>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> || radhe radhe ||
>> >>
>> >> abhishek
>> >
>> >
>> >
>> >
>> > --
>> > || radhe radhe ||
>> >
>> > abhishek
>> >
>> > _______________________________________________
>> > gpaw-users mailing list
>> > gpaw-users at listserv.fysik.dtu.dk
>> > https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>>
>
>
>
> --
> || radhe radhe ||
>
> abhishek
>
--
|| radhe radhe ||
abhishek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.fysik.dtu.dk/pipermail/gpaw-users/attachments/20160120/8d515ec1/attachment-0001.html>
More information about the gpaw-users
mailing list