[ase-users] "Calculator" with client/server connectivity across machines

[Andrew Logsdail]

Hi Ask,

> On 13 May 2020, at 15:17, Ask Hjorth Larsen <asklarsen at gmail.com> wrote:
> 
> Dear Andrew,
> 
> Am Mi., 13. Mai 2020 um 15:29 Uhr schrieb Andrew Logsdail via
> ase-users <ase-users at listserv.fysik.dtu.dk>:
>> 
>> Dear ASE-users,
>> 
>> Recently, we have been encountering issues with system administrators when jobs using ASE have extensive serial compute components compare with the parallel executions (i.e. longer Pythonic part rather than QM energy calculations). The most prevalent example is when coupling with CatLearn, as we find that optimisation on the ML landscape can in some cases become much longer than QM energy calculations used to define the landscape (ML optimisation cost increases with number of data points).
>> 
>> There are several possible ways this could be tackled, and one that came forward in our discussions recently was about running the ASE jobs on a local computer, and defining a calculator object capable of connecting to a remote client through ssh that could then submitting, wait for and analyse jobs, returning necessary data to the host machine for continuation of the scripts (as per the standard ASE calculation). I thought perhaps something like this would already exist but I cannot immediately see functionality that matches the description (the closest I think is the i-Pi sockets, but I think this is designed for the same machine?). Is anyone able to advise if such functionality does exist somewhere? And if not, is there a deliberate reason to avoid this kind of inter-computer connectivity? It’d be something we’d be interesting in investigating further if deemed appropriate.
> 
> The socketio with i-PI protocol works remotely, but it's trickier to
> set up because the initial input file must exist on the compute node
> since the protocol itself only supports changing positions etc.  Feel
> free to tinker with that.
> 
> You can perhaps set it up so ASE talks to another ASE (the socketio
> /client/) remotely, so the remotely running part can generate input
> files.  Then you'd have one remote ASE-ASE socket connection, and one
> local ASE-<computational code> socket connection inside the compute
> node.
> 
> Requires tinkering.
> 
> However the sysadmins probably won't be much happier: The compute
> nodes will still sit still while waiting for the new geometry.  

Yes, this I can still see as the challenge. I was imagining something that would act as a wrapper to a standard ASE calculator but facilitate connect, queue, wait, return. The queue and wait components are key to use of any HPC system, sadly.

> A more
> likely solution is to parallelize or otherwise optimize the learning
> step, as this seems to be the actual bottleneck.

This was one of our other discussed approaches, though is more trickier to achieve as outside my general knowledge area and perhaps a bigger task to achieve. One benefit we see from using the client/server model is it changes the dynamic of work where many QM calculations might be necessary - instead of needing to requeue several times due to e.g. walltime restrictions, one can run continuously on a local server and then submit much shorter jobs on the remote HPC facility.

More thoughts/ideas are welcomed.

Andy

> 
> Best regards
> Ask
> 
>> 
>> Thanks in advance,
>> 
>> Andy
>> 
>> —
>> 
>> Dr. Andrew Logsdail
>> 
>> School of Chemistry,
>> Cardiff University,
>> Main Building,
>> Park Pl,
>> Cardiff CF10 3AT
>> 
>> Yr Ysgol Cemeg,
>> Prifysgol Caerdydd,
>> Y Prif Adeilad,
>> Plas-y-Parc,
>> Caerdydd,
>> CF10 3AT
>> 
>> T/F: +44 2922 510 162
>> E: LogsdailA at cardiff.ac.uk
>> W: https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Flogsdail.github.io%2F&amp;data=02%7C01%7CLogsdailA%40cardiff.ac.uk%7C5d8ca4195a154b4b9ae008d7f748677b%7Cbdb74b3095684856bdbf06759778fcbc%7C1%7C0%7C637249762654250953&amp;sdata=cJ3F2x2uLF%2FGPuhV35pw18UT7F%2B8pdOSBI2jwzQt8sI%3D&amp;reserved=0
>> @: https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fa_logsdail&amp;data=02%7C01%7CLogsdailA%40cardiff.ac.uk%7C5d8ca4195a154b4b9ae008d7f748677b%7Cbdb74b3095684856bdbf06759778fcbc%7C1%7C0%7C637249762654250953&amp;sdata=vd76Hkfb8jFzCIGkEF%2FoL7tTQawA3nG9v1DBdJMcGcQ%3D&amp;reserved=0
>> 
>> The University welcomes correspondence in Welsh or English.
>> Mae'r Brifysgol yn croesawu gohebiaeth yn Gymraeg neu'n Saesneg.
>> 
>> Please note: I sometimes work irregular hours. I don’t expect you to read, action or respond out of your normal work hours.
>> 
>> 
>> _______________________________________________
>> ase-users mailing list
>> ase-users at listserv.fysik.dtu.dk
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flistserv.fysik.dtu.dk%2Fmailman%2Flistinfo%2Fase-users&amp;data=02%7C01%7CLogsdailA%40cardiff.ac.uk%7C5d8ca4195a154b4b9ae008d7f748677b%7Cbdb74b3095684856bdbf06759778fcbc%7C1%7C0%7C637249762654250953&amp;sdata=jdO7LgWgHMigW7n1B4TbtWAtb4Se8dwIAzf2CRrNNt4%3D&amp;reserved=0

— 

Dr. Andrew Logsdail

School of Chemistry, 
Cardiff University, 
Main Building, 
Park Pl, 
Cardiff CF10 3AT 

Yr Ysgol Cemeg,
Prifysgol Caerdydd,
Y Prif Adeilad,
Plas-y-Parc,
Caerdydd,
CF10 3AT

T/F: +44 2922 510 162
E: LogsdailA at cardiff.ac.uk
W: http://logsdail.github.io
@: http://www.twitter.com/a_logsdail

The University welcomes correspondence in Welsh or English.
Mae'r Brifysgol yn croesawu gohebiaeth yn Gymraeg neu'n Saesneg.

Please note: I sometimes work irregular hours. I don’t expect you to read, action or respond out of your normal work hours.