[gpaw-users] MPI-Scalapack library for Python
Jussi Enkovaara
jussi.enkovaara at csc.fi
Thu Sep 28 12:02:52 CEST 2017
On 2017-09-27 12:11, Jens Jørgen Mortensen via gpaw-users wrote:
>
> I agree. In the old days, some supercomputers could not dynamically
> load MPI at runtime and the "make a new Python interpreter" was the only
> way. Today, I think the mpi4py way should be fine. I actually made
> GPAW work with MPI and a standard interpreter some time ago and that
> works fine.
Hi,
there used to be also some MPI implementations where mpirun/mpiexec
added stuff to command line arguments, and MPI_Init then cleaned up the
arguments back to original. Therefore MPI_Init had really be the first
thing to be called after main(), otherwise Python interpreter was
complaining about unknown arguments. Nowadays it should be in most cases
possible to defer MPI_Init when importing mpi module (that is what
happens normally with mpi4py).
Dynamic loading has still some performance handicaps when done in
massive scale ( > 50 000 MPI tasks), and custom interpreter provides
some convenience for debugging and performance analysis, so I would
still keep custom interpreter at least as an option. In any case, that
does not prevent in any way using external Python interfaces e.g. for
Scalapack.
In principle, one could make custom interpreter only with MPI
initialization and without any GPAW C-kernels, and import the C-kernels
dynamically from _gpaw.so, but I think it make sense to include GPAW
kernels statically as built-in if one is in any case building the
interpreter.
(For those not so familiar with custom interpreter, normal Python main
executable looks like (in C-like pseudocode):
main {
Py_Main();
}
and custom interpreter differs mainly as
main {
MPI_Init();
include_gpaw_kernels_in_builtins()
Py_Main()
MPI_Finalize()
}
)
For developing a new Python interface library I would nowadays probably
go with cython. There used to be some issues with big supercomputers and
cross-compilation (that's for example reason that GPAW used to have its
own hdf5 interface), but my recent experiences have been quite positive.
For general purpose scalapack interface I would also rely on getting the
MPI communicator from mpi4py, it should be relatively straightforward to
create a mpi4py communicator object from GPAW's own MPI communicator if
needed.
From pure GPAW perspective one can question a bit the need for general
Python interface to Scalapack. All the (currently) needed Scalapack
routines already have interfaces, and adding new interfaces is
relatively easy. The main bottleneck for some missing parallelism (e.g.
with Davidson eigensolver) is related more to the logic of data
decomposition and not to missing scalapack routines. The same applies in
my opinion to MPI, BLAS, LAPACK: GPAW has stable and good enough (for
its purposes) interface. One should also note that at least some time
ago NumPy/Scipy supported well only very few external BLAS libraries,
while with GPAW it has been always very easy to use any optimized BLAS.
For a Python community in general, Python interface to Scalapack would
of course be quite useful.
Best regards,
Jussi
> Jens Jørgen
>
>> Peter
>>
>> On Wed, Sep 27, 2017 at 10:55 AM, Kuisma, Mikael
>> <mikael.j.kuisma at jyu.fi <mailto:mikael.j.kuisma at jyu.fi>> wrote:
>>
>> Hi!
>>
>> One should consider this a bit. I am not an expert, but I will
>> start with what I know.
>>
>> I am aware of the 'building custom interpreter way', which is
>> relatively easy thing to do (and which I would probably prefer
>> unless I learn something new). This could be possible, that one
>> would always make a custom interpreter which would link to the
>> required modules (mpi, blas, scalapack, gpaw's finite difference
>> toolkit) etc.
>>
>> However, there must be an alternative way? (I mean one does not
>> build a custom interpreter with numpy). Also, in GPAW, in serial
>> mode, it is possible to use the GPAW c-bindings (directly from
>> _gpaw.so).
>>
>> https://gitlab.com/gpaw/gpaw/blob/master/setup.py#L149
>> <https://gitlab.com/gpaw/gpaw/blob/master/setup.py#L149>
>>
>> How does that actually work?
>>
>> Also, why did linking to MPI required custom interpreter (back in
>> the days at least)?
>>
>> BR,
>> Mikael
>>
>>
>> ------------------------------------------------------------------------
>> *Lähettäjä:* gpaw-users-bounces at listserv.fysik.dtu.dk
>> <mailto:gpaw-users-bounces at listserv.fysik.dtu.dk>
>> [gpaw-users-bounces at listserv.fysik.dtu.dk
>> <mailto:gpaw-users-bounces at listserv.fysik.dtu.dk>] käyttäjän marc
>> via gpaw-users [gpaw-users at listserv.fysik.dtu.dk
>> <mailto:gpaw-users at listserv.fysik.dtu.dk>] puolesta
>> *Lähetetty:* 25. syyskuuta 2017 19:15
>> *Vastaanottaja:* gpaw-users at listserv.fysik.dtu.dk
>> <mailto:gpaw-users at listserv.fysik.dtu.dk>; Peter Koval
>>
>> *Aihe:* Re: [gpaw-users] MPI-Scalapack library for Python
>>
>> Hi!
>>
>> I started a repository
>> <https://gitlab.com/mbarbry/python-scalapack> for this project.
>> Since I don't know the Python/C api, I used ctypes for the first
>> implemented function (blacs_pinfo).
>> If some of you want to help me with the Python/C api, as used in
>> GPAW, you are more than welcome, otherwise I will continue with
>> ctypes.
>>
>> Best regards,
>> marc
>>
>> On 09/21/2017 06:00 PM, marc via gpaw-users wrote:
>>> I think for starting we should implement an external library for
>>> Scalapack. But ultimately, I believe that it would be nice for
>>> the community if this functions are available directly through
>>> Scipy, as the BLAS and LAPACK wrapper are already implemented
>>> there. Then, since they are already implemented, I don't think it
>>> is useful to pack BLAS and LAPACK.
>>>
>>> I'm not a GPAW developer, but you could replace the call to BLAS
>>> and LAPACK in GPAW by the scipy wrapper instead of your one. We
>>> are doing it in our own code and the functions are as fast than
>>> C/Fortran implementation (almost all blas routine are
>>> implemented, not so sure for lapack),
>>>
>>> Simple code like this one can be use,
>>>
>>> import scipy.linalg.blas as blas
>>>
>>> C = blas.gemm(1.0, A, B)
>>>
>>> Concerning the MPI wrapper, as Ask already commented, we should
>>> use the mpi4py library, then only Scalapack routines need to be
>>> implemented.
>>>
>>> Marc
>>>
>>> On 09/21/2017 05:42 PM, Kuisma, Mikael wrote:
>>>> Hi!
>>>>
>>>> There is one question which needs to be figured out. As Ask, I
>>>> believe that
>>>> "It would be very useful to have a well-designed stack of Python
>>>> binding for BLAS, LAPACK, MPI, and ScaLAPACK." It could even
>>>> allow new novel parallelization methods and algorithms as they
>>>> become available (say for efficient diagonalization). Such a
>>>> library would in general be quite good to the community even
>>>> outside DFT.
>>>>
>>>> The question is, however, how hard would it be to restructure
>>>> GPAW to use that library? Although, the general library would be
>>>> nice, writing it should make a useful contribution to GPAW
>>>> implementation too. What do other GPAW developers think on the
>>>> tediousness compared to usefullness of packaking BLAS, LAPACK,
>>>> MPI, and ScaLAPACK to one independent package?
>>>>
>>>> Regarding scipy. Why scipy, why not just an external Python
>>>> library with c-bindings?
>>>>
>>>> Mikael
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> *Lähettäjä:* gpaw-users-bounces at listserv.fysik.dtu.dk
>>>> <mailto:gpaw-users-bounces at listserv.fysik.dtu.dk>
>>>> [gpaw-users-bounces at listserv.fysik.dtu.dk
>>>> <mailto:gpaw-users-bounces at listserv.fysik.dtu.dk>] käyttäjän
>>>> Peter Koval via gpaw-users [gpaw-users at listserv.fysik.dtu.dk
>>>> <mailto:gpaw-users at listserv.fysik.dtu.dk>] puolesta
>>>> *Lähetetty:* 21. syyskuuta 2017 17:51
>>>> *Vastaanottaja:* marc Barbry
>>>> *Kopio:* gpaw-users at listserv.fysik.dtu.dk
>>>> <mailto:gpaw-users at listserv.fysik.dtu.dk>
>>>> *Aihe:* Re: [gpaw-users] MPI-Scalapack library for Python
>>>>
>>>> I think scipy could be more tolerant to mpi calls than numpy.
>>>>
>>>> El 21 sept. 2017 4:28 PM, "marc" <marc.barbry at mailoo.org
>>>> <mailto:marc.barbry at mailoo.org>> escribió:
>>>>
>>>> From my experience, scipy has a good parallelization through
>>>> Blas routines if build against the right library (e.g. mkl).
>>>> But this is only for shared memory parallelization and it
>>>> can not be use with MPI. Scipy therefore is missing MPI
>>>> capabilities that it could be nice to add.
>>>>
>>>> On 09/21/2017 04:25 PM, Ask Hjorth Larsen wrote:
>>>>
>>>> It is worth noting that scipy calls a lot of BLAS
>>>> functions, but I
>>>> don't know how it works (maybe they actually have a
>>>> complete
>>>> interface?).
>>>>
>>>> 2017-09-21 16:16 GMT+02:00 Peter Koval via gpaw-users
>>>> <gpaw-users at listserv.fysik.dtu.dk
>>>> <mailto:gpaw-users at listserv.fysik.dtu.dk>>:
>>>>
>>>> Yes, I agree. Ultimately the lib or similar should
>>>> be a pary of scipy.
>>>>
>>>> El 21 sept. 2017 2:50 PM, "marc"
>>>> <marc.barbry at mailoo.org
>>>> <mailto:marc.barbry at mailoo.org>> escribió:
>>>>
>>>> Hi Mikael,
>>>>
>>>> nice to see that you are interested in writing a
>>>> Scalapack wrapper. I
>>>> think it is really missing in python.
>>>> I found one, name scalapy. But it is not
>>>> maintained anymore and the
>>>> installation failed. We could try to write
>>>> something similar using the code
>>>> from GPAW.
>>>>
>>>> Best regards,
>>>> Marc
>>>>
>>>> On 09/21/2017 01:42 PM, Kuisma, Mikael wrote:
>>>>
>>>> Hi!
>>>>
>>>> I added some Scalapack/blacs routines while
>>>> implementing the LCAO-TDDFT
>>>> for GPAW and it was somewhat cumbersome. Also,
>>>> the API we have in GPAW is
>>>> somewhat tedious to use at the moment: we have a
>>>> lot of code to handle
>>>> ScaLAPACK and LAPACK calls as different cases.
>>>> So I would vote for writing a
>>>> general Python-Scalapack library (instead that
>>>> you take code from GPAW to
>>>> your Python/C code). We could make this library
>>>> transparent, so it would
>>>> work also without ScaLAPACK performing serial
>>>> linear algebra..
>>>>
>>>> ScaPACK does not parallize as well as I would
>>>> like it to parallelize.
>>>> With a library wrapper, one could also try other
>>>> libraries (for
>>>> eigensolvers, there are at least more parallel
>>>> solvers than ScaLAPACK).
>>>>
>>>> One more remark regarding GPAW: I always resist
>>>> adding external libraries
>>>> to GPAW, so if one does a python with c-bindings
>>>> library, I think it would
>>>> make sense to embed that Python and C-code
>>>> directly to GPAW repository. The
>>>> end users do not like install new dependencies
>>>> every time the version number
>>>> increases by 0.1. (I wish that we should also
>>>> provide basic working
>>>> installation of libxc and libvdwxc as well in
>>>> our repository, with option to
>>>> link to external libraries).
>>>>
>>>> I do not know about licencing issues that well,
>>>> but as far as I think, it
>>>> should be ok to take code from GPAW (GPL), if
>>>> your code is also GPL and make
>>>> your code available.
>>>>
>>>> BR,
>>>> Mikael Kuisma
>>>>
>>>> Academy of Finland Post Doc
>>>>
>>>> Nanoscience Center
>>>> Department of Chemistry
>>>> University of Jyväskylä
>>>> P.O. Box 35
>>>> FIN-40014 University of Jyväskylä
>>>> Finland
>>>>
>>>> E-mail: mikael.j.kuisma (at) jyu.fi <http://jyu.fi>
>>>> Phone: +358 40 077 4114
>>>> <tel:%2B358%2040%20077%204114>
>>>>
>>>>
>>>> https://scholar.google.se/citations?user=w0eP9esAAAAJ
>>>>
>>>> <https://scholar.google.se/citations?user=w0eP9esAAAAJ>
>>>>
>>>> Visiting address: NSC, 3rd floor, YN320
>>>>
>>>> ________________________________________
>>>> Lähettäjä:
>>>> gpaw-users-bounces at listserv.fysik.dtu.dk
>>>> <mailto:gpaw-users-bounces at listserv.fysik.dtu.dk>
>>>> [gpaw-users-bounces at listserv.fysik.dtu.dk
>>>> <mailto:gpaw-users-bounces at listserv.fysik.dtu.dk>]
>>>> käyttäjän marc via
>>>> gpaw-users [gpaw-users at listserv.fysik.dtu.dk
>>>> <mailto:gpaw-users at listserv.fysik.dtu.dk>] puolesta
>>>> Lähetetty: 21. syyskuuta 2017 14:10
>>>> Vastaanottaja: gpaw-users at listserv.fysik.dtu.dk
>>>> <mailto:gpaw-users at listserv.fysik.dtu.dk>
>>>> Kopio: Peter Koval
>>>> Aihe: [gpaw-users] MPI-Scalapack library for Python
>>>>
>>>> Dear Gpaw developers,
>>>>
>>>> We are actually developing a python code to
>>>> perform efficient TDDFT
>>>> calculation with LCAO. Our program can read DFT
>>>> data from different code
>>>> such as Siesta, OpenMX, Pyscf and GPAW (with
>>>> some limitations for the
>>>> moment).
>>>> We would like to parallelize our code using MPI
>>>> and Scalapack but we
>>>> could not find any library in python that wrap
>>>> both MPI and Scalapack.
>>>> Looking to the GPAW code, we have seen that you
>>>> wrote your own wrapper
>>>> for MPI-Saclapack. My question is the following,
>>>> would it be difficult
>>>> to modify your MPI wrapper to use it in our
>>>> code? Is it feasible from a
>>>> license point of view?
>>>> Do you think it could be feasible to write a
>>>> general python wrapper for
>>>> MPI and Scalapack in an other library using your
>>>> implementation?
>>>>
>>>> Best regards,
>>>> Marc Barbry
>>>> _______________________________________________
>>>> gpaw-users mailing list
>>>> gpaw-users at listserv.fysik.dtu.dk
>>>> <mailto:gpaw-users at listserv.fysik.dtu.dk>
>>>>
>>>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>>>>
>>>> <https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users>
>>>>
>>>>
>>>> _______________________________________________
>>>> gpaw-users mailing list
>>>> gpaw-users at listserv.fysik.dtu.dk
>>>> <mailto:gpaw-users at listserv.fysik.dtu.dk>
>>>>
>>>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>>>>
>>>> <https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users>
>>>>
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> gpaw-users mailing list
>>> gpaw-users at listserv.fysik.dtu.dk
>>> <mailto:gpaw-users at listserv.fysik.dtu.dk>
>>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>>> <https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users>
>>
>>
>>
>>
>> _______________________________________________
>> gpaw-users mailing list
>> gpaw-users at listserv.fysik.dtu.dk
>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>
> _______________________________________________
> gpaw-users mailing list
> gpaw-users at listserv.fysik.dtu.dk
> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
More information about the gpaw-users
mailing list