[gpaw-users] 回复： Assistance Required: Memory Consumption Issue in GPAW Calculations

[Georg Kastlunger]

Dear Wei,
The increase in memo
The fact that writing the trajectory file is to blame is indeed very surprising and should be explored further.
In dependent of that may I suggest you try simplifying the model in two steps:

  1.  Run the same calculation with SolvationGPAW as the calculator instead of SJM. You just have to comment out the “sj”-dictionary in your calculator parameters and it should run.
  2.  Try running with vanilla GPAW as the calculator, where you now also have to comment out the “cavity”, “dielectric” and “interactions” keys.
For both tests monitor the memory consumption and let me know if it behaves differently among tests.

Best wishes,
Georg

From: gpaw-users <gpaw-users-bounces at listserv.fysik.dtu.dk> on behalf of Adil via gpaw-users <gpaw-users at listserv.fysik.dtu.dk>
Date: Thursday, 26 September 2024 at 08.06
To: gpaw-users <gpaw-users at listserv.fysik.dtu.dk>
Subject: [gpaw-users] 回复： Assistance Required: Memory Consumption Issue in GPAW Calculations
Dear all

I hope this email finds you well.

First and foremost, thanks for Jens Jørgen's helpful suggestion.  After addressing the current issue of increasing thread memory consumption over time, I will definitely take your advice into account to optimize the number of cores.

Regarding the memory usage monitoring, I encountered a minor obstacle when using `grep MEM gpaw.out`, as it did not provide the memory usage for each step as expected. To work around this, I utilized `ps -aux | grep gpaw` and `free -h` to monitor the memory usage of gpaw threads and the device. I redirected the outputs of these commands to two separate files: process_mem.txt for gpaw threads and device_mem.txt for device memory. Due to the size exceeding the attachment limit, I have uploaded these files to Google Drive for your convenience. You can access the files using the following link:

https://drive.google.com/drive/folders/1tNg3AiWAYXEIEzQKP8FZnQGLN4dLh0qJ?usp=drive_link

For your immediate reference, I have attached the extracted data files and the plotting script to this email. The attached graph, generated using the data from these files, clearly illustrates the increase in memory usage over time.
[cid:950A890D at 8F946C79.F77FF466.png]
Furthermore, in my recent testing, I noticed that when the trajectory parameter was not specified during the execution of opt.run, the memory usage did not increase as it had previously. I am curious to learn more about the reason for this change.

Thank you once again for your valuable assistance.

Best regards,

Wei Cao

------------------ 原始邮件 ------------------
发件人: "Georg Kastlunger" <geokast at dtu.dk>;
发送时间: 2024年9月24日(星期二) 晚上11:24
收件人: "Jens Jørgen Mortensen"<jjmo at dtu.dk>;"Adil"<weicaocn at qq.com>;"gpaw-users"<gpaw-users at listserv.fysik.dtu.dk>;
抄送: "Peterson, Andrew"<Andrew_Peterson at brown.edu>;"Melander, Marko"<marko.m.melander at jyu.fi>;
主题: Re: [gpaw-users] Assistance Required: Memory Consumption Issue in GPAW Calculations

Dear Wei,

I tried to reproduce your error with your input but couldn’t. The memory usage for me using your input scripts stayed quite constant.
[cid:8C08719B at CEFB6902.F77FF466]

Can you try to produce the same plot, on your machine using the attached input files? The memory usage will be accessible via “grep MEM gpaw.out” after the calculation finished.

Best wishes,
Georg

From:Jens Jørgen Mortensen <jjmo at dtu.dk>
Date: Tuesday, 24 September 2024 at 10.41
To: Adil <weicaocn at qq.com>, gpaw-users <gpaw-users at listserv.fysik.dtu.dk>
Cc: Georg Kastlunger <geokast at dtu.dk>, Peterson, Andrew <Andrew_Peterson at brown.edu>, Melander, Marko <marko.m.melander at jyu.fi>
Subject: Re: [gpaw-users] Assistance Required: Memory Consumption Issue in GPAW Calculations
On 9/21/24 21:19, Adil via gpaw-users wrote:
> Dear gpaw developers
>
> I hope this message finds you well. I am writing to seek your expertise
> regarding an issue I've encountered while performing calculations with
> GPAW.I meet a question about memory consumption recently when I try to
> optimize the geometry of a 80-atoms slab.
>
> The question arise because I find the memory consumption of this task is
> extremely high. When I try to use /--dry-run/ to estimate the memory,
> the anticipated memory consumption was around 500 MiB. However, during
> the actual computation, each process consumed a minimum of 1 GiB of
> memory, and it continued to escalate as the calculation progressed.
> After ten steps of optimization, the memory usage even alarmingly
> reached 7 GiB in some processes. By the end of the computation, it even
> peaked at a single-threaded maximum of 27 GiB.
>
> As for the parallel parameters, I have taken note of the documentation
> which states that if parameters such as domain are not specified, the
> calculator will automatically select an appropriate value. Given this, I
> have decided against manually setting these parameters to avoid any
> potential misconfiguration.
>
> Additionally, I have attached my computation script and output files for
> your comprehensive review.
>
> Could you please provide some guidance on whether this level of memory
> consumption is expected for calculations of this scale and complexity?
> If it is not, I would appreciate any recommendations on how to optimize
> the memory usage or any potential modifications to the calculation
> parameters that might help.
>
> I understand that memory management can be a complex issue, and any
> insights you can offer would be invaluable. If there are any specific
> logs or additional information you need from me to better diagnose the
> problem, please let me know.
>
> Thank you in advance for your time and assistance. I look forward to
> your response.

I'm not so familiar with SJM calculations.  Georg, Andrew, Marko: do you
have any comments?

I can see that you have 5 IBZ k-points, so you could choose a number of
cores that is divisible by 5 - that should reduce memory use.

Jens Jørgen

> Best regards,
> Wei Cao
> Jinan University, China
>
> _______________________________________________
> gpaw-users mailing list
> gpaw-users at listserv.fysik.dtu.dk
> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.fysik.dtu.dk/pipermail/gpaw-users/attachments/20240926/1fa0feb7/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 950A890D at 8F946C79.F77FF466.png
Type: application/octet-stream
Size: 51882 bytes
Desc: 950A890D at 8F946C79.F77FF466.png
URL: <http://listserv.fysik.dtu.dk/pipermail/gpaw-users/attachments/20240926/1fa0feb7/attachment-0001.obj>