[gpaw-users] Increase in process memory for single point calculations
Ask Hjorth Larsen
asklarsen at gmail.com
Tue Aug 21 17:16:22 CEST 2018
2018-08-21 14:00 GMT+02:00 Chang Liu <chang.liu at fysik.su.se>:
> Hi,
>
>
> After adding slab.calc = None at the end of the loop, this is the result for
> 1.3.0:
>
>
> 2co_0_gpaw.out: Process memory now: 81.29 MiB
> 2co_0_gpaw.out- Calculator: 257.25 MiB
> --
> 2co_1_gpaw.out: Process memory now: 361.21 MiB
> 2co_1_gpaw.out- Calculator: 257.25 MiB
> --
> 2co_2_gpaw.out: Process memory now: 369.78 MiB
> 2co_2_gpaw.out- Calculator: 273.71 MiB
> --
> 2co_3_gpaw.out: Process memory now: 391.88 MiB
> 2co_3_gpaw.out- Calculator: 273.69 MiB
> --
> 2co_4_gpaw.out: Process memory now: 398.46 MiB
> 2co_4_gpaw.out- Calculator: 273.71 MiB
> --
> 2co_5_gpaw.out: Process memory now: 400.30 MiB
> 2co_5_gpaw.out- Calculator: 273.70 MiB
> --
> 2co_6_gpaw.out: Process memory now: 407.50 MiB
> 2co_6_gpaw.out- Calculator: 257.25 MiB
> --
> 2co_7_gpaw.out: Process memory now: 407.50 MiB
> 2co_7_gpaw.out- Calculator: 257.25 MiB
> --
> 2co_8_gpaw.out: Process memory now: 407.50 MiB
> 2co_8_gpaw.out- Calculator: 273.58 MiB
> --
> 2co_9_gpaw.out: Process memory now: 410.55 MiB
> 2co_9_gpaw.out- Calculator: 273.46 MiB
> --
> 2co_10_gpaw.out: Process memory now: 410.55 MiB
> 2co_10_gpaw.out- Calculator: 273.43 MiB
> --
> 2co_11_gpaw.out: Process memory now: 410.61 MiB
> 2co_11_gpaw.out- Calculator: 273.42 MiB
> --
> 2co_12_gpaw.out: Process memory now: 412.32 MiB
> 2co_12_gpaw.out- Calculator: 273.51 MiB
> --
> 2co_13_gpaw.out: Process memory now: 412.74 MiB
> 2co_13_gpaw.out- Calculator: 273.43 MiB
> --
> 2co_14_gpaw.out: Process memory now: 419.50 MiB
> 2co_14_gpaw.out- Calculator: 273.38 MiB
> --
> 2co_15_gpaw.out: Process memory now: 423.67 MiB
> 2co_15_gpaw.out- Calculator: 281.86 MiB
> --
> 2co_16_gpaw.out: Process memory now: 432.04 MiB
> 2co_16_gpaw.out- Calculator: 281.85 MiB
> --
> 2co_17_gpaw.out: Process memory now: 438.82 MiB
> 2co_17_gpaw.out- Calculator: 281.84 MiB
> --
> 2co_18_gpaw.out: Process memory now: 438.82 MiB
> 2co_18_gpaw.out- Calculator: 281.85 MiB
> --
> 2co_19_gpaw.out: Process memory now: 445.02 MiB
> 2co_19_gpaw.out- Calculator: 281.85 MiB
> --
> 2co_20_gpaw.out: Process memory now: 449.26 MiB
> 2co_20_gpaw.out- Calculator: 281.85 MiB
> --
> 2co_21_gpaw.out: Process memory now: 453.70 MiB
> 2co_21_gpaw.out- Calculator: 281.82 MiB
>
> And this is the result for the 1.4.0 version:
>
>
> 2co_0_gpaw.out: Process memory now: 94.17 MiB
> 2co_0_gpaw.out- Calculator: 183.75 MiB
> --
> 2co_1_gpaw.out: Process memory now: 415.62 MiB
> 2co_1_gpaw.out- Calculator: 183.75 MiB
> --
> 2co_2_gpaw.out: Process memory now: 425.35 MiB
> 2co_2_gpaw.out- Calculator: 194.96 MiB
> --
> 2co_3_gpaw.out: Process memory now: 461.36 MiB
> 2co_3_gpaw.out- Calculator: 194.95 MiB
> --
> 2co_4_gpaw.out: Process memory now: 461.36 MiB
> 2co_4_gpaw.out- Calculator: 194.96 MiB
> --
> 2co_5_gpaw.out: Process memory now: 461.36 MiB
> 2co_5_gpaw.out- Calculator: 194.96 MiB
> --
> 2co_6_gpaw.out: Process memory now: 461.36 MiB
> 2co_6_gpaw.out- Calculator: 183.75 MiB
> --
> 2co_7_gpaw.out: Process memory now: 461.36 MiB
> 2co_7_gpaw.out- Calculator: 183.75 MiB
> --
> 2co_8_gpaw.out: Process memory now: 461.36 MiB
> 2co_8_gpaw.out- Calculator: 194.83 MiB
> --
> 2co_9_gpaw.out: Process memory now: 461.36 MiB
> 2co_9_gpaw.out- Calculator: 194.71 MiB
> --
> 2co_10_gpaw.out: Process memory now: 466.47 MiB
> 2co_10_gpaw.out- Calculator: 194.69 MiB
> --
> 2co_11_gpaw.out: Process memory now: 466.85 MiB
> 2co_11_gpaw.out- Calculator: 194.68 MiB
> --
> 2co_12_gpaw.out: Process memory now: 477.61 MiB
> 2co_12_gpaw.out- Calculator: 194.76 MiB
> --
> 2co_13_gpaw.out: Process memory now: 478.95 MiB
> 2co_13_gpaw.out- Calculator: 194.69 MiB
> --
> 2co_14_gpaw.out: Process memory now: 483.20 MiB
> 2co_14_gpaw.out- Calculator: 194.64 MiB
> --
> 2co_15_gpaw.out: Process memory now: 484.82 MiB
> 2co_15_gpaw.out- Calculator: 200.50 MiB
> --
> 2co_16_gpaw.out: Process memory now: 504.63 MiB
> 2co_16_gpaw.out- Calculator: 200.49 MiB
> --
> 2co_17_gpaw.out: Process memory now: 509.45 MiB
> 2co_17_gpaw.out- Calculator: 200.48 MiB
> --
> 2co_18_gpaw.out: Process memory now: 510.81 MiB
> 2co_18_gpaw.out- Calculator: 200.49 MiB
> --
> 2co_19_gpaw.out: Process memory now: 522.61 MiB
> 2co_19_gpaw.out- Calculator: 200.49 MiB
> --
> 2co_20_gpaw.out: Process memory now: 525.21 MiB
> 2co_20_gpaw.out- Calculator: 200.49 MiB
> --
> 2co_21_gpaw.out: Process memory now: 526.09 MiB
> 2co_21_gpaw.out- Calculator: 200.46 MiB
>
> The calculations did not crash this time. I think the problem is more or
> less solved since now the increase is very small. Thank you very much for
> the help!
Good! Although it may be ugly that memory still increases, this
commonly happens when the computer still has memory left memory and
allocates new buffers dynamically. It should stabilize eventually
when there is enough memory that it always finds contiguous chunks for
all the arrays.
Best regards
Ask
>
>
> Best wishes,
> Chang Liu
> ----------------------------------------------------------
> Chang Liu
> PhD student
> Fysikum, Albanova, Stockholm University
> S-106 91 Stockholm, Sweden
> +46 767159891
> http://xsolasgroup.fysik.su.se/
> ________________________________
> From: Jens Jørgen Mortensen <jjmo at dtu.dk>
> Sent: Friday, August 17, 2018 10:26:08 AM
> To: Chang Liu; Ask Hjorth Larsen
>
> Cc: gpaw-users at listserv.fysik.dtu.dk
> Subject: Re: [gpaw-users] Increase in process memory for single point
> calculations
>
> Den 17-08-2018 kl. 10:05 skrev Chang Liu via gpaw-users:
>
> Hi,
>
>
> I have reinstalled GPAW-1.4.0 with MPI/2.1.1 and did a test with a
> trajectory file of 22 images. The input is:
>
>
> from gpaw import *
> from ase.io import *
> from gpaw.utilities import h2gpts
> import numpy as np
>
>
> mixer = Mixer(0.1, 5, weight = 100.0)
>
> def calc():
> return GPAW(poissonsolver = {'dipolelayer':'xy','eps':1e-12},
> kpts = {'density':2},
> xc = 'RPBE',
> symmetry = {'point_group':False},
> mixer = mixer,
> eigensolver=Davidson(3),
> spinpol = False,
> occupations = FermiDirac(0.1),
> maxiter = 1000,
>
> )
> neb_states = read('neb_2co.traj@:')
>
> for slab in neb_states:
> slab.center(vacuum=5.,axis=2)
> slab.cell[2][2]+=10.
> slab.pbc = (1,1,0)
> slab.set_calculator(calc())
> slab.calc.set(gpts = h2gpts(0.2, slab.get_cell(), idiv=8),txt =
> '2co_%d_gpaw.out' % neb_states.index(slab))
> e_f = slab.get_potential_energy()
> f = open('Energy_2co_%d_gpaw.out' % neb_states.index(slab), 'w')
> f.write('%.3f\n' % e_f)
> f.close()
>
>
> Try to add "slab.calc = None" as the last line of the loop.
>
> Jens Jørgen
>
>
> The memory usage for 1.3.0 is:
>
>
> b-an01 [~/pfs/co2rr/2018/180815/1.3.0]$ grep -r --include='*out' -i 'process
> memory' -A 1 $(ls -tr *)
> 2co_0_gpaw.out: Process memory now: 80.99 MiB
> 2co_0_gpaw.out- Calculator: 257.25 MiB
> --
> 2co_1_gpaw.out: Process memory now: 361.00 MiB
> 2co_1_gpaw.out- Calculator: 257.25 MiB
> --
> 2co_2_gpaw.out: Process memory now: 619.15 MiB
> 2co_2_gpaw.out- Calculator: 273.71 MiB
> --
> 2co_3_gpaw.out: Process memory now: 889.65 MiB
> 2co_3_gpaw.out- Calculator: 273.69 MiB
> --
> 2co_4_gpaw.out: Process memory now: 1167.81 MiB
> 2co_4_gpaw.out- Calculator: 273.71 MiB
> --
> 2co_5_gpaw.out: Process memory now: 1441.73 MiB
> 2co_5_gpaw.out- Calculator: 273.70 MiB
> --
> 2co_6_gpaw.out: Process memory now: 1717.00 MiB
> 2co_6_gpaw.out- Calculator: 257.25 MiB
> --
> 2co_7_gpaw.out: Process memory now: 1977.18 MiB
> 2co_7_gpaw.out- Calculator: 257.25 MiB
> --
> 2co_8_gpaw.out: Process memory now: 2222.76 MiB
> 2co_8_gpaw.out- Calculator: 273.58 MiB
> --
> 2co_9_gpaw.out: Process memory now: 2495.62 MiB
> 2co_9_gpaw.out- Calculator: 273.46 MiB
> --
> 2co_10_gpaw.out: Process memory now: 2769.93 MiB
> 2co_10_gpaw.out- Calculator: 273.43 MiB
> --
> 2co_11_gpaw.out: Process memory now: 3049.25 MiB
> 2co_11_gpaw.out- Calculator: 273.42 MiB
> --
> 2co_12_gpaw.out: Process memory now: 3323.22 MiB
> 2co_12_gpaw.out- Calculator: 273.51 MiB
> --
> 2co_13_gpaw.out: Process memory now: 3599.27 MiB
> 2co_13_gpaw.out- Calculator: 273.43 MiB
> --
> 2co_14_gpaw.out: Process memory now: 3876.67 MiB
> 2co_14_gpaw.out- Calculator: 273.38 MiB
> --
> 2co_15_gpaw.out: Process memory now: 4150.99 MiB
> 2co_15_gpaw.out- Calculator: 281.86 MiB
> --
> 2co_16_gpaw.out: Process memory now: 4434.88 MiB
> 2co_16_gpaw.out- Calculator: 281.85 MiB
> --
> 2co_18_gpaw.out: Process memory now: 5011.75 MiB
> 2co_18_gpaw.out- Calculator: 281.85 MiB
> --
> 2co_17_gpaw.out: Process memory now: 4722.66 MiB
> 2co_17_gpaw.out- Calculator: 281.84 MiB
>
> For 1.4.0, it is:
>
>
> b-an01 [~/pfs/co2rr/2018/180815/1.4.0]$ grep -r --include='*out' -i 'process
> memory' -A 1 $(ls -tr *)
> 2co_0_gpaw.out: Process memory now: 94.30 MiB
> 2co_0_gpaw.out- Calculator: 183.75 MiB
> --
> 2co_1_gpaw.out: Process memory now: 415.34 MiB
> 2co_1_gpaw.out- Calculator: 183.75 MiB
> --
> 2co_2_gpaw.out: Process memory now: 675.71 MiB
> 2co_2_gpaw.out- Calculator: 194.96 MiB
> --
> 2co_3_gpaw.out: Process memory now: 954.99 MiB
> 2co_3_gpaw.out- Calculator: 194.95 MiB
> --
> 2co_4_gpaw.out: Process memory now: 1222.41 MiB
> 2co_4_gpaw.out- Calculator: 194.96 MiB
> --
> 2co_5_gpaw.out: Process memory now: 1493.82 MiB
> 2co_5_gpaw.out- Calculator: 194.96 MiB
> --
> 2co_6_gpaw.out: Process memory now: 1760.62 MiB
> 2co_6_gpaw.out- Calculator: 183.75 MiB
> --
> 2co_7_gpaw.out: Process memory now: 2018.56 MiB
> 2co_7_gpaw.out- Calculator: 183.75 MiB
> --
> 2co_8_gpaw.out: Process memory now: 2280.84 MiB
> 2co_8_gpaw.out- Calculator: 194.83 MiB
> --
> 2co_9_gpaw.out: Process memory now: 2553.50 MiB
> 2co_9_gpaw.out- Calculator: 194.71 MiB
> --
> 2co_10_gpaw.out: Process memory now: 2827.13 MiB
> 2co_10_gpaw.out- Calculator: 194.69 MiB
> --
> 2co_11_gpaw.out: Process memory now: 3105.79 MiB
> 2co_11_gpaw.out- Calculator: 194.68 MiB
> --
> 2co_12_gpaw.out: Process memory now: 3385.24 MiB
> 2co_12_gpaw.out- Calculator: 194.76 MiB
> --
> 2co_13_gpaw.out: Process memory now: 3664.50 MiB
> 2co_13_gpaw.out- Calculator: 194.69 MiB
> --
> 2co_14_gpaw.out: Process memory now: 3937.29 MiB
> 2co_14_gpaw.out- Calculator: 194.64 MiB
> --
> 2co_15_gpaw.out: Process memory now: 4212.07 MiB
> 2co_15_gpaw.out- Calculator: 200.50 MiB
> --
> 2co_16_gpaw.out: Process memory now: 4498.34 MiB
> 2co_16_gpaw.out- Calculator: 200.49 MiB
> --
> 2co_17_gpaw.out: Process memory now: 4786.58 MiB
> 2co_17_gpaw.out- Calculator: 200.48 MiB
> --
> 2co_18_gpaw.out: Process memory now: 5068.21 MiB
> 2co_18_gpaw.out- Calculator: 200.49 MiB
> --
> 2co_19_gpaw.out: Process memory now: 5344.71 MiB
> 2co_19_gpaw.out- Calculator: 200.49 MiB
> --
> 2co_20_gpaw.out: Process memory now: 5614.39 MiB
> 2co_20_gpaw.out- Calculator: 200.49 MiB
>
> The calculation crashes when the memory use grows to the maximum in both
> cases.
>
>
> Could I avoid this by reading in one image into "slab" each time in the loop
> instead? Thanks.
>
>
> Best wishes,
> Chang Liu
> ----------------------------------------------------------
> Chang Liu
> PhD student
> Fysikum, Albanova, Stockholm University
> S-106 91 Stockholm, Sweden
> +46 767159891
> http://xsolasgroup.fysik.su.se/
> ________________________________
> From: Ask Hjorth Larsen <asklarsen at gmail.com>
> Sent: Wednesday, August 15, 2018 10:45:10 PM
> To: Chang Liu
> Cc: gpaw-users at listserv.fysik.dtu.dk
> Subject: Re: [gpaw-users] Increase in process memory for single point
> calculations
>
> Dear Chang Liu,
>
> 2018-08-15 7:24 GMT-05:00 Chang Liu via gpaw-users
> <gpaw-users at listserv.fysik.dtu.dk>:
>> Hi GPAW users,
>>
>>
>> I have encountered a strange problem of increasing process memory when I
>> did
>> a few single point calculations in loop. Here is my input file:
>>
>>
>> from gpaw import *
>> from ase.io import *
>> from gpaw.utilities import h2gpts
>> import numpy as np
>>
>>
>> names = ['2co','6co']
>> states = ['is','ts','fs']
>> tsind = {'2co':6,'6co':4}
>>
>> mixer = Mixer(0.1, 5, weight = 100.0)
>>
>> def calc():
>> return GPAW(poissonsolver = {'dipolelayer':'xy','eps':1e-12},
>> kpts = {'density':2},
>> xc = 'RPBE',
>> symmetry = {'point_group':False},
>> mixer = mixer,
>> eigensolver=Davidson(3),
>> spinpol = False,
>> occupations = FermiDirac(0.1),
>> maxiter = 1000,
>>
>> )
>> for name in names:
>> neb_states = read('../%s/neb_all.traj@:'%name)
>> for state in states:
>> if state == 'is': slab = neb_states[0]
>> if state == 'fs': slab = neb_states[-1]
>> else: slab = neb_states[tsind[name]]
>> slab.center(vacuum=5.,axis=2)
>> slab.cell[2][2]+=10.
>> slab.pbc = (1,1,0)
>> slab.set_calculator(calc())
>> slab.calc.set(gpts = h2gpts(0.2, slab.get_cell(), idiv=8),txt =
>> '%s_%s_gpaw.out' % (name,state))
>> e_f = slab.get_potential_energy()
>> f = open('Energy_%s_%s_gpaw.out' % (name,state), 'w')
>> f.write('%.3f\n' % e_f)
>> f.close()
>
> There will be at least 3 coexisting calculators because neb_states
> holds three atoms objects, each with a distinct calculator. So the
> increase in memory is not surprising.
>
> In fact what I don't understand is why it does not increase *always*,
> at least for the first three steps. After that you would expect a few
> small increases because it randomly claims a bit more heap space, but
> should stabilize.
>
> But the printouts do not disprove this. The loops would need to be
> much longer to see if there is a problem.
>
> Best regards
> Ask
>
>
>>
>> For GPAW-1.3.0, the output files for these calculations shows:
>>
>>
>> b-an01 [~/pfs/co2rr/2018/180812/1.3.0]$ grep -r --include='*out' -i
>> 'process
>> memory' -A 1 $(ls -tr *)
>> 2co_is_gpaw.out: Process memory now: 80.45 MiB
>> 2co_is_gpaw.out- Calculator: 148.19 MiB
>> --
>> 2co_ts_gpaw.out: Process memory now: 255.31 MiB
>> 2co_ts_gpaw.out- Calculator: 148.19 MiB
>> --
>> 2co_fs_gpaw.out: Process memory now: 256.38 MiB
>> 2co_fs_gpaw.out- Calculator: 156.32 MiB
>> --
>> 6co_is_gpaw.out: Process memory now: 414.14 MiB
>> 6co_is_gpaw.out- Calculator: 169.99 MiB
>> --
>> 6co_ts_gpaw.out: Process memory now: 414.14 MiB
>> 6co_ts_gpaw.out- Calculator: 169.99 MiB
>> --
>> 6co_fs_gpaw.out: Process memory now: 414.14 MiB
>> 6co_fs_gpaw.out- Calculator: 170.05 MiB
>>
>> For GPAW-1.4.0, it shows:
>>
>>
>> b-an01 [~/pfs/co2rr/2018/180812/1.4.0]$ grep -r --include='*out' -i
>> 'process
>> memory' -A 1 $(ls -tr *)
>> 2co_is_gpaw.out: Process memory now: 103.06 MiB
>> 2co_is_gpaw.out- Calculator: 108.73 MiB
>> --
>> 2co_ts_gpaw.out: Process memory now: 353.70 MiB
>> 2co_ts_gpaw.out- Calculator: 108.73 MiB
>> --
>> 2co_fs_gpaw.out: Process memory now: 394.64 MiB
>> 2co_fs_gpaw.out- Calculator: 114.23 MiB
>> --
>> 6co_is_gpaw.out: Process memory now: 589.16 MiB
>> 6co_is_gpaw.out- Calculator: 124.54 MiB
>> --
>> 6co_ts_gpaw.out: Process memory now: 589.16 MiB
>> 6co_ts_gpaw.out- Calculator: 124.54 MiB
>> --
>> 6co_fs_gpaw.out: Process memory now: 606.05 MiB
>> 6co_fs_gpaw.out- Calculator: 124.60 MiB
>>
>> It becomes a severe problem when I tried to do more single point
>> calculations with different potentials with the Solvated Jellium Model
>> calculator since in that case, the growth of process memory goes too fast
>> so
>> that the program crashes. Could you help me? Thank you very much!
>>
>>
>> Best wishes,
>> Chang Liu
>> ----------------------------------------------------------
>> Chang Liu
>> PhD student
>> Fysikum, Albanova, Stockholm University
>> S-106 91 Stockholm, Sweden
>> +46 767159891
>> http://xsolasgroup.fysik.su.se/
>>
>> _______________________________________________
>> gpaw-users mailing list
>> gpaw-users at listserv.fysik.dtu.dk
>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>
>
> _______________________________________________
> gpaw-users mailing list
> gpaw-users at listserv.fysik.dtu.dk
> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>
>
More information about the gpaw-users
mailing list