[gpaw-users] Suggestions for Optimizing GPAW with GPU Acceleration

[Jens Jørgen Mortensen]

On 9/14/24 04:24, David Mauricio Mosquera Narvaez via gpaw-users wrote:
> Dear GPAW community,
> 
> I hope this message finds you well.
> 
> I’m excited to share that I have successfully installed GPAW with GPU 
> acceleration and have noticed significant performance improvements in my 
> calculations. However, I’m reaching out to ask if you have any 
> additional tips or best practices for further optimizing the speed of my 
> computations, particularly for GPU acceleration.
> 
> I've been exploring ways to improve the performance and came across the 
> possibility of making adjustments in the |potential.py| file. 
> Specifically, I modified the |get_vacuum_level| function as follows:
> 
> 
>     def get_vacuum_level(self) -> float:
>          grid = self.vt_sR.desc
>          if grid.pbc_c.all():
>              return np.nan
>          if grid.zerobc_c.any():
>              return 0.0
>          if self.vHt_x is None:
>              raise ValueError('No electrostatic potential')
> 
>          # Handling UGArray and PWArray cases
>          if isinstance(self.vHt_x, UGArray):
>              vHt_r = self.vHt_x.gather()
>          elif isinstance(self.vHt_x, PWArray):
>              vHt_g = self.vHt_x.gather()
>              if vHt_g is not None:
>                  vHt_g = cp.asarray(vHt_g)  # Convert to CuPy array for 
> GPU operations
> 
>                  # Use CuPy's GPU-accelerated inverse FFT
>                  vHt_r = cp.fft.ifft(vHt_g)
>              else:
>                  vHt_r = None
>          else:
>              return np.nan  # TB-mode
> 
>          vacuum_level = 0.0
>          if vHt_r is not None:
>              # Ensure we handle both NumPy and CuPy arrays
>              if hasattr(vHt_r.data, 'get'):
>                  # CuPy array, convert to NumPy
>                  vHt_r_data = vHt_r.data
>              else:
>                   # NumPy array, convert to CuPy for GPU operations
>                  vHt_r_data = cp.asarray(vHt_r.data)
> 
>              # Iterate over the periodic directions
>              for c, periodic in enumerate(grid.pbc_c):
>                  if not periodic:
>                      # Use CuPy functions for GPU computations
>                      vacuum_level += cp.asnumpy(cp.moveaxis(vHt_r_data, 
> c, 0)[0].mean())
>              vacuum_level /= (3 - grid.pbc_c.sum())
> 
>          # Convert to a scalar value
>          vacuum_level_scalar = float(vacuum_level)  # Ensure it's a scalar
> 
>          # Print the scalar value instead of logging
>          print(f'vacuum-level: {vacuum_level_scalar:.3f}  # V')
> 
>          return vacuum_level_scalar * Ha
> 
> 
> 
> This modification leverages CuPy for GPU-accelerated FFT operations and 
> ensures efficient handling of both NumPy and CuPy arrays. However, I am 
> open to further improvements or alternate approaches to fully maximize 
> GPU utilization.
> 
> If you have any suggestions for enhancing performance or insights on 
> best practices for using GPUs with GPAW, I would greatly appreciate your 
> input.
> 
> Thank you for your time, and I look forward to your feedback!

Great to hear that you got GPAW going on a GPU.  It's still early days 
for the GPU-code and only a certain things are supported:

https://gpaw.readthedocs.io/documentation/gpu.html

About your "get_vacuum_level()" changes:  There was a bug in that part 
of the code that would make GPU-calculations crash.  It's been fixed now:

https://gitlab.com/gpaw/gpaw/-/merge_requests/2475

We are very interested in hearing about bugs, code that is too slow or 
things not implemented.  Improvements are very welcome.  Preferably in a 
branch on GitLab (makes it easier to see the changes).

Jens Jørgen

> 
> Best regards,
> 
> 
> David Mauricio
> 
> 
> 
> 
> *Aviso legal:* El contenido de este mensaje y los archivos adjuntos son 
> confidenciales y de uso exclusivo de la Universidad Nacional de 
> Colombia. Se encuentran dirigidos sólo para el uso del destinatario al 
> cual van enviados. La reproducción, lectura y/o copia se encuentran 
> prohibidas a cualquier persona diferente a este y puede ser ilegal. Si 
> usted lo ha recibido por error, infórmenos y elimínelo de su correo. Los 
> Datos Personales serán tratados conforme a la Ley 1581 de 2012 y a 
> nuestra Política de Datos Personales que podrá consultar en la página 
> web www.unal.edu.co <http://www.unal.edu.co/>.__Las opiniones, 
> informaciones, conclusiones y cualquier otro tipo de dato contenido en 
> este correo electrónico, no relacionados con la actividad de la 
> Universidad Nacional de Colombia, se entenderá como personales y de 
> ninguna manera son avaladas por la Universidad.
> 
> _______________________________________________
> gpaw-users mailing list
> gpaw-users at listserv.fysik.dtu.dk
> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users