[gpaw-users] MPI-Scalapack library for Python
Jens Jørgen Mortensen
jjmo at dtu.dk
Wed Sep 27 11:11:53 CEST 2017
On 09/27/2017 10:59 AM, Peter Koval via gpaw-users wrote:
> Well, if you know how to build a custom interpreter, then you are the
> expert. However, I vote for a simpler experience like current scipy
> provides: one can use functions from a normal python. If it is
> possible in mpi2py, it should be possible for scalapack as well.
>
I agree. In the old days, some supercomputers could not dynamically
load MPI at runtime and the "make a new Python interpreter" was the only
way. Today, I think the mpi4py way should be fine. I actually made
GPAW work with MPI and a standard interpreter some time ago and that
works fine.
Jens Jørgen
> Peter
>
> On Wed, Sep 27, 2017 at 10:55 AM, Kuisma, Mikael
> <mikael.j.kuisma at jyu.fi <mailto:mikael.j.kuisma at jyu.fi>> wrote:
>
> Hi!
>
> One should consider this a bit. I am not an expert, but I will
> start with what I know.
>
> I am aware of the 'building custom interpreter way', which is
> relatively easy thing to do (and which I would probably prefer
> unless I learn something new). This could be possible, that one
> would always make a custom interpreter which would link to the
> required modules (mpi, blas, scalapack, gpaw's finite difference
> toolkit) etc.
>
> However, there must be an alternative way? (I mean one does not
> build a custom interpreter with numpy). Also, in GPAW, in serial
> mode, it is possible to use the GPAW c-bindings (directly from
> _gpaw.so).
>
> https://gitlab.com/gpaw/gpaw/blob/master/setup.py#L149
> <https://gitlab.com/gpaw/gpaw/blob/master/setup.py#L149>
>
> How does that actually work?
>
> Also, why did linking to MPI required custom interpreter (back in
> the days at least)?
>
> BR,
> Mikael
>
> ------------------------------------------------------------------------
> *Lähettäjä:* gpaw-users-bounces at listserv.fysik.dtu.dk
> <mailto:gpaw-users-bounces at listserv.fysik.dtu.dk>
> [gpaw-users-bounces at listserv.fysik.dtu.dk
> <mailto:gpaw-users-bounces at listserv.fysik.dtu.dk>] käyttäjän marc
> via gpaw-users [gpaw-users at listserv.fysik.dtu.dk
> <mailto:gpaw-users at listserv.fysik.dtu.dk>] puolesta
> *Lähetetty:* 25. syyskuuta 2017 19:15
> *Vastaanottaja:* gpaw-users at listserv.fysik.dtu.dk
> <mailto:gpaw-users at listserv.fysik.dtu.dk>; Peter Koval
>
> *Aihe:* Re: [gpaw-users] MPI-Scalapack library for Python
>
> Hi!
>
> I started a repository
> <https://gitlab.com/mbarbry/python-scalapack> for this project.
> Since I don't know the Python/C api, I used ctypes for the first
> implemented function (blacs_pinfo).
> If some of you want to help me with the Python/C api, as used in
> GPAW, you are more than welcome, otherwise I will continue with
> ctypes.
>
> Best regards,
> marc
>
> On 09/21/2017 06:00 PM, marc via gpaw-users wrote:
>> I think for starting we should implement an external library for
>> Scalapack. But ultimately, I believe that it would be nice for
>> the community if this functions are available directly through
>> Scipy, as the BLAS and LAPACK wrapper are already implemented
>> there. Then, since they are already implemented, I don't think it
>> is useful to pack BLAS and LAPACK.
>>
>> I'm not a GPAW developer, but you could replace the call to BLAS
>> and LAPACK in GPAW by the scipy wrapper instead of your one. We
>> are doing it in our own code and the functions are as fast than
>> C/Fortran implementation (almost all blas routine are
>> implemented, not so sure for lapack),
>>
>> Simple code like this one can be use,
>>
>> import scipy.linalg.blas as blas
>>
>> C = blas.gemm(1.0, A, B)
>>
>> Concerning the MPI wrapper, as Ask already commented, we should
>> use the mpi4py library, then only Scalapack routines need to be
>> implemented.
>>
>> Marc
>>
>> On 09/21/2017 05:42 PM, Kuisma, Mikael wrote:
>>> Hi!
>>>
>>> There is one question which needs to be figured out. As Ask, I
>>> believe that
>>> "It would be very useful to have a well-designed stack of Python
>>> binding for BLAS, LAPACK, MPI, and ScaLAPACK." It could even
>>> allow new novel parallelization methods and algorithms as they
>>> become available (say for efficient diagonalization). Such a
>>> library would in general be quite good to the community even
>>> outside DFT.
>>>
>>> The question is, however, how hard would it be to restructure
>>> GPAW to use that library? Although, the general library would be
>>> nice, writing it should make a useful contribution to GPAW
>>> implementation too. What do other GPAW developers think on the
>>> tediousness compared to usefullness of packaking BLAS, LAPACK,
>>> MPI, and ScaLAPACK to one independent package?
>>>
>>> Regarding scipy. Why scipy, why not just an external Python
>>> library with c-bindings?
>>>
>>> Mikael
>>>
>>> ------------------------------------------------------------------------
>>> *Lähettäjä:* gpaw-users-bounces at listserv.fysik.dtu.dk
>>> <mailto:gpaw-users-bounces at listserv.fysik.dtu.dk>
>>> [gpaw-users-bounces at listserv.fysik.dtu.dk
>>> <mailto:gpaw-users-bounces at listserv.fysik.dtu.dk>] käyttäjän
>>> Peter Koval via gpaw-users [gpaw-users at listserv.fysik.dtu.dk
>>> <mailto:gpaw-users at listserv.fysik.dtu.dk>] puolesta
>>> *Lähetetty:* 21. syyskuuta 2017 17:51
>>> *Vastaanottaja:* marc Barbry
>>> *Kopio:* gpaw-users at listserv.fysik.dtu.dk
>>> <mailto:gpaw-users at listserv.fysik.dtu.dk>
>>> *Aihe:* Re: [gpaw-users] MPI-Scalapack library for Python
>>>
>>> I think scipy could be more tolerant to mpi calls than numpy.
>>>
>>> El 21 sept. 2017 4:28 PM, "marc" <marc.barbry at mailoo.org
>>> <mailto:marc.barbry at mailoo.org>> escribió:
>>>
>>> From my experience, scipy has a good parallelization through
>>> Blas routines if build against the right library (e.g. mkl).
>>> But this is only for shared memory parallelization and it
>>> can not be use with MPI. Scipy therefore is missing MPI
>>> capabilities that it could be nice to add.
>>>
>>> On 09/21/2017 04:25 PM, Ask Hjorth Larsen wrote:
>>>
>>> It is worth noting that scipy calls a lot of BLAS
>>> functions, but I
>>> don't know how it works (maybe they actually have a complete
>>> interface?).
>>>
>>> 2017-09-21 16:16 GMT+02:00 Peter Koval via gpaw-users
>>> <gpaw-users at listserv.fysik.dtu.dk
>>> <mailto:gpaw-users at listserv.fysik.dtu.dk>>:
>>>
>>> Yes, I agree. Ultimately the lib or similar should
>>> be a pary of scipy.
>>>
>>> El 21 sept. 2017 2:50 PM, "marc"
>>> <marc.barbry at mailoo.org
>>> <mailto:marc.barbry at mailoo.org>> escribió:
>>>
>>> Hi Mikael,
>>>
>>> nice to see that you are interested in writing a
>>> Scalapack wrapper. I
>>> think it is really missing in python.
>>> I found one, name scalapy. But it is not
>>> maintained anymore and the
>>> installation failed. We could try to write
>>> something similar using the code
>>> from GPAW.
>>>
>>> Best regards,
>>> Marc
>>>
>>> On 09/21/2017 01:42 PM, Kuisma, Mikael wrote:
>>>
>>> Hi!
>>>
>>> I added some Scalapack/blacs routines while
>>> implementing the LCAO-TDDFT
>>> for GPAW and it was somewhat cumbersome. Also,
>>> the API we have in GPAW is
>>> somewhat tedious to use at the moment: we have a
>>> lot of code to handle
>>> ScaLAPACK and LAPACK calls as different cases.
>>> So I would vote for writing a
>>> general Python-Scalapack library (instead that
>>> you take code from GPAW to
>>> your Python/C code). We could make this library
>>> transparent, so it would
>>> work also without ScaLAPACK performing serial
>>> linear algebra..
>>>
>>> ScaPACK does not parallize as well as I would
>>> like it to parallelize.
>>> With a library wrapper, one could also try other
>>> libraries (for
>>> eigensolvers, there are at least more parallel
>>> solvers than ScaLAPACK).
>>>
>>> One more remark regarding GPAW: I always resist
>>> adding external libraries
>>> to GPAW, so if one does a python with c-bindings
>>> library, I think it would
>>> make sense to embed that Python and C-code
>>> directly to GPAW repository. The
>>> end users do not like install new dependencies
>>> every time the version number
>>> increases by 0.1. (I wish that we should also
>>> provide basic working
>>> installation of libxc and libvdwxc as well in
>>> our repository, with option to
>>> link to external libraries).
>>>
>>> I do not know about licencing issues that well,
>>> but as far as I think, it
>>> should be ok to take code from GPAW (GPL), if
>>> your code is also GPL and make
>>> your code available.
>>>
>>> BR,
>>> Mikael Kuisma
>>>
>>> Academy of Finland Post Doc
>>>
>>> Nanoscience Center
>>> Department of Chemistry
>>> University of Jyväskylä
>>> P.O. Box 35
>>> FIN-40014 University of Jyväskylä
>>> Finland
>>>
>>> E-mail: mikael.j.kuisma (at) jyu.fi <http://jyu.fi>
>>> Phone: +358 40 077 4114
>>> <tel:%2B358%2040%20077%204114>
>>>
>>> https://scholar.google.se/citations?user=w0eP9esAAAAJ
>>> <https://scholar.google.se/citations?user=w0eP9esAAAAJ>
>>>
>>> Visiting address: NSC, 3rd floor, YN320
>>>
>>> ________________________________________
>>> Lähettäjä:
>>> gpaw-users-bounces at listserv.fysik.dtu.dk
>>> <mailto:gpaw-users-bounces at listserv.fysik.dtu.dk>
>>> [gpaw-users-bounces at listserv.fysik.dtu.dk
>>> <mailto:gpaw-users-bounces at listserv.fysik.dtu.dk>]
>>> käyttäjän marc via
>>> gpaw-users [gpaw-users at listserv.fysik.dtu.dk
>>> <mailto:gpaw-users at listserv.fysik.dtu.dk>] puolesta
>>> Lähetetty: 21. syyskuuta 2017 14:10
>>> Vastaanottaja: gpaw-users at listserv.fysik.dtu.dk
>>> <mailto:gpaw-users at listserv.fysik.dtu.dk>
>>> Kopio: Peter Koval
>>> Aihe: [gpaw-users] MPI-Scalapack library for Python
>>>
>>> Dear Gpaw developers,
>>>
>>> We are actually developing a python code to
>>> perform efficient TDDFT
>>> calculation with LCAO. Our program can read DFT
>>> data from different code
>>> such as Siesta, OpenMX, Pyscf and GPAW (with
>>> some limitations for the
>>> moment).
>>> We would like to parallelize our code using MPI
>>> and Scalapack but we
>>> could not find any library in python that wrap
>>> both MPI and Scalapack.
>>> Looking to the GPAW code, we have seen that you
>>> wrote your own wrapper
>>> for MPI-Saclapack. My question is the following,
>>> would it be difficult
>>> to modify your MPI wrapper to use it in our
>>> code? Is it feasible from a
>>> license point of view?
>>> Do you think it could be feasible to write a
>>> general python wrapper for
>>> MPI and Scalapack in an other library using your
>>> implementation?
>>>
>>> Best regards,
>>> Marc Barbry
>>> _______________________________________________
>>> gpaw-users mailing list
>>> gpaw-users at listserv.fysik.dtu.dk
>>> <mailto:gpaw-users at listserv.fysik.dtu.dk>
>>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>>> <https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users>
>>>
>>>
>>> _______________________________________________
>>> gpaw-users mailing list
>>> gpaw-users at listserv.fysik.dtu.dk
>>> <mailto:gpaw-users at listserv.fysik.dtu.dk>
>>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>>> <https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users>
>>>
>>>
>>
>>
>>
>> _______________________________________________
>> gpaw-users mailing list
>> gpaw-users at listserv.fysik.dtu.dk
>> <mailto:gpaw-users at listserv.fysik.dtu.dk>
>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>> <https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users>
>
>
>
>
> _______________________________________________
> gpaw-users mailing list
> gpaw-users at listserv.fysik.dtu.dk
> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
More information about the gpaw-users
mailing list