[gpaw-users] Increase in process memory for single point calculations
Chang Liu
chang.liu at fysik.su.se
Tue Aug 21 14:00:19 CEST 2018
Hi,
After adding slab.calc = None at the end of the loop, this is the result for 1.3.0:
2co_0_gpaw.out: Process memory now: 81.29 MiB
2co_0_gpaw.out- Calculator: 257.25 MiB
--
2co_1_gpaw.out: Process memory now: 361.21 MiB
2co_1_gpaw.out- Calculator: 257.25 MiB
--
2co_2_gpaw.out: Process memory now: 369.78 MiB
2co_2_gpaw.out- Calculator: 273.71 MiB
--
2co_3_gpaw.out: Process memory now: 391.88 MiB
2co_3_gpaw.out- Calculator: 273.69 MiB
--
2co_4_gpaw.out: Process memory now: 398.46 MiB
2co_4_gpaw.out- Calculator: 273.71 MiB
--
2co_5_gpaw.out: Process memory now: 400.30 MiB
2co_5_gpaw.out- Calculator: 273.70 MiB
--
2co_6_gpaw.out: Process memory now: 407.50 MiB
2co_6_gpaw.out- Calculator: 257.25 MiB
--
2co_7_gpaw.out: Process memory now: 407.50 MiB
2co_7_gpaw.out- Calculator: 257.25 MiB
--
2co_8_gpaw.out: Process memory now: 407.50 MiB
2co_8_gpaw.out- Calculator: 273.58 MiB
--
2co_9_gpaw.out: Process memory now: 410.55 MiB
2co_9_gpaw.out- Calculator: 273.46 MiB
--
2co_10_gpaw.out: Process memory now: 410.55 MiB
2co_10_gpaw.out- Calculator: 273.43 MiB
--
2co_11_gpaw.out: Process memory now: 410.61 MiB
2co_11_gpaw.out- Calculator: 273.42 MiB
--
2co_12_gpaw.out: Process memory now: 412.32 MiB
2co_12_gpaw.out- Calculator: 273.51 MiB
--
2co_13_gpaw.out: Process memory now: 412.74 MiB
2co_13_gpaw.out- Calculator: 273.43 MiB
--
2co_14_gpaw.out: Process memory now: 419.50 MiB
2co_14_gpaw.out- Calculator: 273.38 MiB
--
2co_15_gpaw.out: Process memory now: 423.67 MiB
2co_15_gpaw.out- Calculator: 281.86 MiB
--
2co_16_gpaw.out: Process memory now: 432.04 MiB
2co_16_gpaw.out- Calculator: 281.85 MiB
--
2co_17_gpaw.out: Process memory now: 438.82 MiB
2co_17_gpaw.out- Calculator: 281.84 MiB
--
2co_18_gpaw.out: Process memory now: 438.82 MiB
2co_18_gpaw.out- Calculator: 281.85 MiB
--
2co_19_gpaw.out: Process memory now: 445.02 MiB
2co_19_gpaw.out- Calculator: 281.85 MiB
--
2co_20_gpaw.out: Process memory now: 449.26 MiB
2co_20_gpaw.out- Calculator: 281.85 MiB
--
2co_21_gpaw.out: Process memory now: 453.70 MiB
2co_21_gpaw.out- Calculator: 281.82 MiB
And this is the result for the 1.4.0 version:
2co_0_gpaw.out: Process memory now: 94.17 MiB
2co_0_gpaw.out- Calculator: 183.75 MiB
--
2co_1_gpaw.out: Process memory now: 415.62 MiB
2co_1_gpaw.out- Calculator: 183.75 MiB
--
2co_2_gpaw.out: Process memory now: 425.35 MiB
2co_2_gpaw.out- Calculator: 194.96 MiB
--
2co_3_gpaw.out: Process memory now: 461.36 MiB
2co_3_gpaw.out- Calculator: 194.95 MiB
--
2co_4_gpaw.out: Process memory now: 461.36 MiB
2co_4_gpaw.out- Calculator: 194.96 MiB
--
2co_5_gpaw.out: Process memory now: 461.36 MiB
2co_5_gpaw.out- Calculator: 194.96 MiB
--
2co_6_gpaw.out: Process memory now: 461.36 MiB
2co_6_gpaw.out- Calculator: 183.75 MiB
--
2co_7_gpaw.out: Process memory now: 461.36 MiB
2co_7_gpaw.out- Calculator: 183.75 MiB
--
2co_8_gpaw.out: Process memory now: 461.36 MiB
2co_8_gpaw.out- Calculator: 194.83 MiB
--
2co_9_gpaw.out: Process memory now: 461.36 MiB
2co_9_gpaw.out- Calculator: 194.71 MiB
--
2co_10_gpaw.out: Process memory now: 466.47 MiB
2co_10_gpaw.out- Calculator: 194.69 MiB
--
2co_11_gpaw.out: Process memory now: 466.85 MiB
2co_11_gpaw.out- Calculator: 194.68 MiB
--
2co_12_gpaw.out: Process memory now: 477.61 MiB
2co_12_gpaw.out- Calculator: 194.76 MiB
--
2co_13_gpaw.out: Process memory now: 478.95 MiB
2co_13_gpaw.out- Calculator: 194.69 MiB
--
2co_14_gpaw.out: Process memory now: 483.20 MiB
2co_14_gpaw.out- Calculator: 194.64 MiB
--
2co_15_gpaw.out: Process memory now: 484.82 MiB
2co_15_gpaw.out- Calculator: 200.50 MiB
--
2co_16_gpaw.out: Process memory now: 504.63 MiB
2co_16_gpaw.out- Calculator: 200.49 MiB
--
2co_17_gpaw.out: Process memory now: 509.45 MiB
2co_17_gpaw.out- Calculator: 200.48 MiB
--
2co_18_gpaw.out: Process memory now: 510.81 MiB
2co_18_gpaw.out- Calculator: 200.49 MiB
--
2co_19_gpaw.out: Process memory now: 522.61 MiB
2co_19_gpaw.out- Calculator: 200.49 MiB
--
2co_20_gpaw.out: Process memory now: 525.21 MiB
2co_20_gpaw.out- Calculator: 200.49 MiB
--
2co_21_gpaw.out: Process memory now: 526.09 MiB
2co_21_gpaw.out- Calculator: 200.46 MiB
The calculations did not crash this time. I think the problem is more or less solved since now the increase is very small. Thank you very much for the help!
Best wishes,
Chang Liu
----------------------------------------------------------
Chang Liu
PhD student
Fysikum, Albanova, Stockholm University
S-106 91 Stockholm, Sweden
+46 767159891
http://xsolasgroup.fysik.su.se/
________________________________
From: Jens Jørgen Mortensen <jjmo at dtu.dk>
Sent: Friday, August 17, 2018 10:26:08 AM
To: Chang Liu; Ask Hjorth Larsen
Cc: gpaw-users at listserv.fysik.dtu.dk
Subject: Re: [gpaw-users] Increase in process memory for single point calculations
Den 17-08-2018 kl. 10:05 skrev Chang Liu via gpaw-users:
Hi,
I have reinstalled GPAW-1.4.0 with MPI/2.1.1 and did a test with a trajectory file of 22 images. The input is:
from gpaw import *
from ase.io import *
from gpaw.utilities import h2gpts
import numpy as np
mixer = Mixer(0.1, 5, weight = 100.0)
def calc():
return GPAW(poissonsolver = {'dipolelayer':'xy','eps':1e-12},
kpts = {'density':2},
xc = 'RPBE',
symmetry = {'point_group':False},
mixer = mixer,
eigensolver=Davidson(3),
spinpol = False,
occupations = FermiDirac(0.1),
maxiter = 1000,
)
neb_states = read('neb_2co.traj@:')
for slab in neb_states:
slab.center(vacuum=5.,axis=2)
slab.cell[2][2]+=10.
slab.pbc = (1,1,0)
slab.set_calculator(calc())
slab.calc.set(gpts = h2gpts(0.2, slab.get_cell(), idiv=8),txt = '2co_%d_gpaw.out' % neb_states.index(slab))
e_f = slab.get_potential_energy()
f = open('Energy_2co_%d_gpaw.out' % neb_states.index(slab), 'w')
f.write('%.3f\n' % e_f)
f.close()
Try to add "slab.calc = None" as the last line of the loop.
Jens Jørgen
The memory usage for 1.3.0 is:
b-an01 [~/pfs/co2rr/2018/180815/1.3.0]$ grep -r --include='*out' -i 'process memory' -A 1 $(ls -tr *)
2co_0_gpaw.out: Process memory now: 80.99 MiB
2co_0_gpaw.out- Calculator: 257.25 MiB
--
2co_1_gpaw.out: Process memory now: 361.00 MiB
2co_1_gpaw.out- Calculator: 257.25 MiB
--
2co_2_gpaw.out: Process memory now: 619.15 MiB
2co_2_gpaw.out- Calculator: 273.71 MiB
--
2co_3_gpaw.out: Process memory now: 889.65 MiB
2co_3_gpaw.out- Calculator: 273.69 MiB
--
2co_4_gpaw.out: Process memory now: 1167.81 MiB
2co_4_gpaw.out- Calculator: 273.71 MiB
--
2co_5_gpaw.out: Process memory now: 1441.73 MiB
2co_5_gpaw.out- Calculator: 273.70 MiB
--
2co_6_gpaw.out: Process memory now: 1717.00 MiB
2co_6_gpaw.out- Calculator: 257.25 MiB
--
2co_7_gpaw.out: Process memory now: 1977.18 MiB
2co_7_gpaw.out- Calculator: 257.25 MiB
--
2co_8_gpaw.out: Process memory now: 2222.76 MiB
2co_8_gpaw.out- Calculator: 273.58 MiB
--
2co_9_gpaw.out: Process memory now: 2495.62 MiB
2co_9_gpaw.out- Calculator: 273.46 MiB
--
2co_10_gpaw.out: Process memory now: 2769.93 MiB
2co_10_gpaw.out- Calculator: 273.43 MiB
--
2co_11_gpaw.out: Process memory now: 3049.25 MiB
2co_11_gpaw.out- Calculator: 273.42 MiB
--
2co_12_gpaw.out: Process memory now: 3323.22 MiB
2co_12_gpaw.out- Calculator: 273.51 MiB
--
2co_13_gpaw.out: Process memory now: 3599.27 MiB
2co_13_gpaw.out- Calculator: 273.43 MiB
--
2co_14_gpaw.out: Process memory now: 3876.67 MiB
2co_14_gpaw.out- Calculator: 273.38 MiB
--
2co_15_gpaw.out: Process memory now: 4150.99 MiB
2co_15_gpaw.out- Calculator: 281.86 MiB
--
2co_16_gpaw.out: Process memory now: 4434.88 MiB
2co_16_gpaw.out- Calculator: 281.85 MiB
--
2co_18_gpaw.out: Process memory now: 5011.75 MiB
2co_18_gpaw.out- Calculator: 281.85 MiB
--
2co_17_gpaw.out: Process memory now: 4722.66 MiB
2co_17_gpaw.out- Calculator: 281.84 MiB
For 1.4.0, it is:
b-an01 [~/pfs/co2rr/2018/180815/1.4.0]$ grep -r --include='*out' -i 'process memory' -A 1 $(ls -tr *)
2co_0_gpaw.out: Process memory now: 94.30 MiB
2co_0_gpaw.out- Calculator: 183.75 MiB
--
2co_1_gpaw.out: Process memory now: 415.34 MiB
2co_1_gpaw.out- Calculator: 183.75 MiB
--
2co_2_gpaw.out: Process memory now: 675.71 MiB
2co_2_gpaw.out- Calculator: 194.96 MiB
--
2co_3_gpaw.out: Process memory now: 954.99 MiB
2co_3_gpaw.out- Calculator: 194.95 MiB
--
2co_4_gpaw.out: Process memory now: 1222.41 MiB
2co_4_gpaw.out- Calculator: 194.96 MiB
--
2co_5_gpaw.out: Process memory now: 1493.82 MiB
2co_5_gpaw.out- Calculator: 194.96 MiB
--
2co_6_gpaw.out: Process memory now: 1760.62 MiB
2co_6_gpaw.out- Calculator: 183.75 MiB
--
2co_7_gpaw.out: Process memory now: 2018.56 MiB
2co_7_gpaw.out- Calculator: 183.75 MiB
--
2co_8_gpaw.out: Process memory now: 2280.84 MiB
2co_8_gpaw.out- Calculator: 194.83 MiB
--
2co_9_gpaw.out: Process memory now: 2553.50 MiB
2co_9_gpaw.out- Calculator: 194.71 MiB
--
2co_10_gpaw.out: Process memory now: 2827.13 MiB
2co_10_gpaw.out- Calculator: 194.69 MiB
--
2co_11_gpaw.out: Process memory now: 3105.79 MiB
2co_11_gpaw.out- Calculator: 194.68 MiB
--
2co_12_gpaw.out: Process memory now: 3385.24 MiB
2co_12_gpaw.out- Calculator: 194.76 MiB
--
2co_13_gpaw.out: Process memory now: 3664.50 MiB
2co_13_gpaw.out- Calculator: 194.69 MiB
--
2co_14_gpaw.out: Process memory now: 3937.29 MiB
2co_14_gpaw.out- Calculator: 194.64 MiB
--
2co_15_gpaw.out: Process memory now: 4212.07 MiB
2co_15_gpaw.out- Calculator: 200.50 MiB
--
2co_16_gpaw.out: Process memory now: 4498.34 MiB
2co_16_gpaw.out- Calculator: 200.49 MiB
--
2co_17_gpaw.out: Process memory now: 4786.58 MiB
2co_17_gpaw.out- Calculator: 200.48 MiB
--
2co_18_gpaw.out: Process memory now: 5068.21 MiB
2co_18_gpaw.out- Calculator: 200.49 MiB
--
2co_19_gpaw.out: Process memory now: 5344.71 MiB
2co_19_gpaw.out- Calculator: 200.49 MiB
--
2co_20_gpaw.out: Process memory now: 5614.39 MiB
2co_20_gpaw.out- Calculator: 200.49 MiB
The calculation crashes when the memory use grows to the maximum in both cases.
Could I avoid this by reading in one image into "slab" each time in the loop instead? Thanks.
Best wishes,
Chang Liu
----------------------------------------------------------
Chang Liu
PhD student
Fysikum, Albanova, Stockholm University
S-106 91 Stockholm, Sweden
+46 767159891
http://xsolasgroup.fysik.su.se/
________________________________
From: Ask Hjorth Larsen <asklarsen at gmail.com><mailto:asklarsen at gmail.com>
Sent: Wednesday, August 15, 2018 10:45:10 PM
To: Chang Liu
Cc: gpaw-users at listserv.fysik.dtu.dk<mailto:gpaw-users at listserv.fysik.dtu.dk>
Subject: Re: [gpaw-users] Increase in process memory for single point calculations
Dear Chang Liu,
2018-08-15 7:24 GMT-05:00 Chang Liu via gpaw-users
<gpaw-users at listserv.fysik.dtu.dk><mailto:gpaw-users at listserv.fysik.dtu.dk>:
> Hi GPAW users,
>
>
> I have encountered a strange problem of increasing process memory when I did
> a few single point calculations in loop. Here is my input file:
>
>
> from gpaw import *
> from ase.io import *
> from gpaw.utilities import h2gpts
> import numpy as np
>
>
> names = ['2co','6co']
> states = ['is','ts','fs']
> tsind = {'2co':6,'6co':4}
>
> mixer = Mixer(0.1, 5, weight = 100.0)
>
> def calc():
> return GPAW(poissonsolver = {'dipolelayer':'xy','eps':1e-12},
> kpts = {'density':2},
> xc = 'RPBE',
> symmetry = {'point_group':False},
> mixer = mixer,
> eigensolver=Davidson(3),
> spinpol = False,
> occupations = FermiDirac(0.1),
> maxiter = 1000,
>
> )
> for name in names:
> neb_states = read('../%s/neb_all.traj@:'%name)
> for state in states:
> if state == 'is': slab = neb_states[0]
> if state == 'fs': slab = neb_states[-1]
> else: slab = neb_states[tsind[name]]
> slab.center(vacuum=5.,axis=2)
> slab.cell[2][2]+=10.
> slab.pbc = (1,1,0)
> slab.set_calculator(calc())
> slab.calc.set(gpts = h2gpts(0.2, slab.get_cell(), idiv=8),txt =
> '%s_%s_gpaw.out' % (name,state))
> e_f = slab.get_potential_energy()
> f = open('Energy_%s_%s_gpaw.out' % (name,state), 'w')
> f.write('%.3f\n' % e_f)
> f.close()
There will be at least 3 coexisting calculators because neb_states
holds three atoms objects, each with a distinct calculator. So the
increase in memory is not surprising.
In fact what I don't understand is why it does not increase *always*,
at least for the first three steps. After that you would expect a few
small increases because it randomly claims a bit more heap space, but
should stabilize.
But the printouts do not disprove this. The loops would need to be
much longer to see if there is a problem.
Best regards
Ask
>
> For GPAW-1.3.0, the output files for these calculations shows:
>
>
> b-an01 [~/pfs/co2rr/2018/180812/1.3.0]$ grep -r --include='*out' -i 'process
> memory' -A 1 $(ls -tr *)
> 2co_is_gpaw.out: Process memory now: 80.45 MiB
> 2co_is_gpaw.out- Calculator: 148.19 MiB
> --
> 2co_ts_gpaw.out: Process memory now: 255.31 MiB
> 2co_ts_gpaw.out- Calculator: 148.19 MiB
> --
> 2co_fs_gpaw.out: Process memory now: 256.38 MiB
> 2co_fs_gpaw.out- Calculator: 156.32 MiB
> --
> 6co_is_gpaw.out: Process memory now: 414.14 MiB
> 6co_is_gpaw.out- Calculator: 169.99 MiB
> --
> 6co_ts_gpaw.out: Process memory now: 414.14 MiB
> 6co_ts_gpaw.out- Calculator: 169.99 MiB
> --
> 6co_fs_gpaw.out: Process memory now: 414.14 MiB
> 6co_fs_gpaw.out- Calculator: 170.05 MiB
>
> For GPAW-1.4.0, it shows:
>
>
> b-an01 [~/pfs/co2rr/2018/180812/1.4.0]$ grep -r --include='*out' -i 'process
> memory' -A 1 $(ls -tr *)
> 2co_is_gpaw.out: Process memory now: 103.06 MiB
> 2co_is_gpaw.out- Calculator: 108.73 MiB
> --
> 2co_ts_gpaw.out: Process memory now: 353.70 MiB
> 2co_ts_gpaw.out- Calculator: 108.73 MiB
> --
> 2co_fs_gpaw.out: Process memory now: 394.64 MiB
> 2co_fs_gpaw.out- Calculator: 114.23 MiB
> --
> 6co_is_gpaw.out: Process memory now: 589.16 MiB
> 6co_is_gpaw.out- Calculator: 124.54 MiB
> --
> 6co_ts_gpaw.out: Process memory now: 589.16 MiB
> 6co_ts_gpaw.out- Calculator: 124.54 MiB
> --
> 6co_fs_gpaw.out: Process memory now: 606.05 MiB
> 6co_fs_gpaw.out- Calculator: 124.60 MiB
>
> It becomes a severe problem when I tried to do more single point
> calculations with different potentials with the Solvated Jellium Model
> calculator since in that case, the growth of process memory goes too fast so
> that the program crashes. Could you help me? Thank you very much!
>
>
> Best wishes,
> Chang Liu
> ----------------------------------------------------------
> Chang Liu
> PhD student
> Fysikum, Albanova, Stockholm University
> S-106 91 Stockholm, Sweden
> +46 767159891
> http://xsolasgroup.fysik.su.se/
>
> _______________________________________________
> gpaw-users mailing list
> gpaw-users at listserv.fysik.dtu.dk<mailto:gpaw-users at listserv.fysik.dtu.dk>
> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
_______________________________________________
gpaw-users mailing list
gpaw-users at listserv.fysik.dtu.dk<mailto:gpaw-users at listserv.fysik.dtu.dk>
https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.fysik.dtu.dk/pipermail/gpaw-users/attachments/20180821/ca41bb5e/attachment-0001.html>
More information about the gpaw-users
mailing list