[gpaw-users] 回复：回复： Assistance Required: Memory Consumption Issue in GPAW Calculations

Sat Sep 28 08:08:52 CEST 2024

Dear Georg

Just wanted to give you an update—I've run those two tests you suggested, but it looks like the memory usage is still creeping up over time. I have uploaded the output file of the calculation results and the graph of memory consumption over time to Google Drive for your review.

https://drive.google.com/file/d/1tXolYCkRMqXlpDR8lgif7xPtbD_9fqMT/view?usp=drive_link

Let me know if you need any more information.

Best regards,

Wei Cao

------------------ 原始邮件 ------------------
发件人:                                                                                                                        "Georg Kastlunger"                                                                                    <geokast at dtu.dk>;
发送时间: 2024年9月26日(星期四) 下午2:30
收件人: "Adil"<weicaocn at qq.com>;"gpaw-users"<gpaw-users at listserv.fysik.dtu.dk>;

主题: Re: [gpaw-users] 回复：  Assistance Required: Memory Consumption Issue in GPAW Calculations

Dear Wei,

The increase in memo

The fact that writing the trajectory file is to blame is indeed very surprising and should be explored further. 

In dependent of that may I suggest you try simplifying the model in two steps:

Run the same calculation with SolvationGPAW as the calculator instead of SJM. You just have to comment out the  “sj”-dictionary in your calculator parameters and it should run. 

Try running with vanilla GPAW as the calculator, where you now also have to comment out the “cavity”, “dielectric”  and “interactions” keys.

For both tests monitor the memory consumption and let me know if it behaves differently among tests. 

Best wishes,

Georg 

From: gpaw-users <gpaw-users-bounces at listserv.fysik.dtu.dk> on behalf of Adil via gpaw-users <gpaw-users at listserv.fysik.dtu.dk>
 Date: Thursday, 26 September 2024 at 08.06
 To: gpaw-users <gpaw-users at listserv.fysik.dtu.dk>
 Subject: [gpaw-users] 回复： Assistance Required: Memory Consumption Issue in GPAW Calculations

Dear all

I hope this email finds you well.

First and foremost, thanks for Jens Jørgen's helpful suggestion.  After addressing the current issue of increasing thread memory consumption over time, I will  definitely take your advice into account to optimize the number of cores.

Regarding the memory usage monitoring, I encountered a minor obstacle when using `grep MEM gpaw.out`, as it did not provide the memory usage for each step as  expected. To work around this, I utilized `ps -aux | grep gpaw` and `free -h` to monitor the memory usage of gpaw threads and the device. I redirected the outputs of these commands to two separate files: process_mem.txt for gpaw threads and device_mem.txt  for device memory. Due to the size exceeding the attachment limit, I have uploaded these files to Google Drive for your convenience. You can access the files using the following link: 

https://drive.google.com/drive/folders/1tNg3AiWAYXEIEzQKP8FZnQGLN4dLh0qJ?usp=drive_link

For your immediate reference, I have attached the extracted data files and the plotting script to this email. The attached graph, generated using the data from  these files, clearly illustrates the increase in memory usage over time.

Furthermore, in my recent testing, I noticed that when the trajectory parameter was not specified during the execution of opt.run, the memory usage did not increase  as it had previously. I am curious to learn more about the reason for this change.

Thank you once again for your valuable assistance.

Best regards,

Wei Cao

------------------ 原始邮件  ------------------

发件人:  "Georg Kastlunger" <geokast at dtu.dk>;

发送时间: 2024年9月24日(星期二) 晚上11:24

收件人: "Jens Jørgen  Mortensen"<jjmo at dtu.dk>;"Adil"<weicaocn at qq.com>;"gpaw-users"<gpaw-users at listserv.fysik.dtu.dk>;

抄送: "Peterson, Andrew"<Andrew_Peterson at brown.edu>;"Melander,  Marko"<marko.m.melander at jyu.fi>;

主题: Re:  [gpaw-users] Assistance Required: Memory Consumption Issue in GPAW Calculations

Dear Wei,

I tried to reproduce your error with your input but couldn’t. The memory usage for me using your input scripts stayed quite constant.

Can you try to produce the same plot, on your machine using the attached input files? The memory usage will be accessible via “grep MEM  gpaw.out” after the calculation finished.

Best wishes,

Georg

From:Jens Jørgen Mortensen <jjmo at dtu.dk>
 Date: Tuesday, 24 September 2024 at 10.41
 To: Adil <weicaocn at qq.com>, gpaw-users <gpaw-users at listserv.fysik.dtu.dk>
 Cc: Georg Kastlunger <geokast at dtu.dk>, Peterson, Andrew <Andrew_Peterson at brown.edu>, Melander, Marko <marko.m.melander at jyu.fi>
 Subject: Re: [gpaw-users] Assistance Required: Memory Consumption Issue in GPAW Calculations

On 9/21/24 21:19, Adil via gpaw-users wrote:
 > Dear gpaw developers
 > 
 > I hope this message finds you well. I am writing to seek your expertise 
 > regarding an issue I've encountered while performing calculations with 
 > GPAW.I meet a question about memory consumption recently when I try to 
 > optimize the geometry of a 80-atoms slab.
 > 
 > The question arise because I find the memory consumption of this task is 
 > extremely high. When I try to use /--dry-run/ to estimate the memory, 
 > the anticipated memory consumption was around 500 MiB. However, during 
 > the actual computation, each process consumed a minimum of 1 GiB of 
 > memory, and it continued to escalate as the calculation progressed. 
 > After ten steps of optimization, the memory usage even alarmingly 
 > reached 7 GiB in some processes. By the end of the computation, it even 
 > peaked at a single-threaded maximum of 27 GiB.
 > 
 > As for the parallel parameters, I have taken note of the documentation 
 > which states that if parameters such as domain are not specified, the 
 > calculator will automatically select an appropriate value. Given this, I 
 > have decided against manually setting these parameters to avoid any 
 > potential misconfiguration.
 > 
 > Additionally, I have attached my computation script and output files for 
 > your comprehensive review.
 > 
 > Could you please provide some guidance on whether this level of memory 
 > consumption is expected for calculations of this scale and complexity? 
 > If it is not, I would appreciate any recommendations on how to optimize 
 > the memory usage or any potential modifications to the calculation 
 > parameters that might help.
 > 
 > I understand that memory management can be a complex issue, and any 
 > insights you can offer would be invaluable. If there are any specific 
 > logs or additional information you need from me to better diagnose the 
 > problem, please let me know.
 > 
 > Thank you in advance for your time and assistance. I look forward to 
 > your response.

 I'm not so familiar with SJM calculations.  Georg, Andrew, Marko: do you 
 have any comments?

 I can see that you have 5 IBZ k-points, so you could choose a number of 
 cores that is divisible by 5 - that should reduce memory use.

 Jens Jørgen

 > Best regards,
 > Wei Cao
 > Jinan University, China
 > 
 > _______________________________________________
 > gpaw-users mailing list
 > gpaw-users at listserv.fysik.dtu.dk
 > https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.fysik.dtu.dk/pipermail/gpaw-users/attachments/20240928/a15d15a1/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 496EED3D at 27B02439.749DF766.jpg
Type: image/jpeg
Size: 51882 bytes
Desc: not available
URL: <http://listserv.fysik.dtu.dk/pipermail/gpaw-users/attachments/20240928/a15d15a1/attachment-0001.jpg>

[gpaw-users] 回复： 回复： Assistance Required: Memory Consumption Issue in GPAW Calculations

[gpaw-users] 回复：回复： Assistance Required: Memory Consumption Issue in GPAW Calculations