[ase-users] Opinion about potentially adding Cython as a dependency

[Ask Hjorth Larsen]

2017-10-25 14:54 GMT+02:00 Jussi Enkovaara via ase-users
<ase-users at listserv.fysik.dtu.dk>:
> Hi,
> generally, I think that Cython is nowadays quite useful for speeding up
> Python code, and if starting a new project from scratch I might even
> consider using Cython for some computational heavy kernels. However, I just
> want to point out that allowing ASE code to depend on Cython is pretty much
> equivalent of allowing C extensions in ASE. I guess having Cython as
> optional dependency might be fine, but as said, one should then accept
> optionally also C-code within ASE.

I agree: It is a compilation step, so from an
installation/distribution perspective (the most important perspective
IMHO) it is equivalent to C extensions.

Somewhat related: It would be fun to see what happens in GPAW if some
modules are cythonized.  It might remove a lot of overhead here and
there, and it does not represent such a fundamental change because
there's a compilation step anyway.

Best regards
Ask

>
> Best regards,
> Jussi
>
>
> On 2017-10-17 10:22, Jens Jørgen Mortensen via ase-users wrote:
>>
>> On 10/12/2017 04:01 PM, Gaël Donval wrote:
>>>>
>>>> If it can be done on a wholly opt-in basis then I think it is great.
>>>> The question of C extensions and dependencies on compiled libraries
>>>> have come up several times because they have legitimate applications,
>>>> but anything that makes installation trickier will also affect users
>>>> who don't need those features.  And then there's the question of
>>>> release testing (platform dependence).
>>>
>>> I suspect releasing wheels will solve the problem for most of us
>>> anyway. Apparently, people usually take advantage CI systems to
>>> generate theirs but I have not tried it myself.
>>>
>>> So my take out of this it: it is fine to use Cython provided that its
>>> absence (or the absence of working compiler) doesn't affect people who
>>> are not using the functionality. So I guess, at best, fall-back
>>> function should be provided or, if really impractical, no failure
>>> should occur as long as the specific functionalities are not used.
>>>
>>>> Can we establish a working minimal example where ASE builds and uses
>>>> Cython to perform some trivial action, and see how it affects the
>>>> build requiements?
>>>
>>> I don't have time to do that just now but I can implement existing
>>> functions in, say, ase/geometry/distance.py in cython this week-end. It
>>> would be small and decoupled enough not to be too invasive and yet
>>> provide a useful sandbox.
>>
>>
>> My preference would be to write C-code which most people can
>> read/understand/maintain and then write the interface to Python using
>> cython/cffi/ctypes.  And only do this if a Python+NumPy+SciPy solution is 10
>> times slower or uses 10 times more memory.
>>
>> Jens Jørgen
>>
>>> Cheers,
>>> Gaël
>>>
>>>
>>>> Best regards
>>>> Ask
>>>>
>>>> 2017-10-11 6:47 GMT+02:00 Jens Jørgen Mortensen <jjmo at dtu.dk>:
>>>>>
>>>>> Den 10-10-2017 kl. 16:32 skrev Gaël Donval via ase-users:
>>>>>
>>>>> Dear Ask,
>>>>>
>>>>> Dear Gaël,
>>>>>
>>>>> 2017-10-10 12:52 GMT+02:00 Gaël Donval via ase-users
>>>>> <ase-users at listserv.fysik.dtu.dk>:
>>>>>
>>>>> Dear list,
>>>>>
>>>>> I am currently working on some geometrical code that I will
>>>>> ultimately
>>>>> propose adding to ASE. I am currently using Cython to speed up what
>>>>> needs speeding up and, to a more limited extent, I also use it to
>>>>> very
>>>>> conveniently generate bindings to C.
>>>>>
>>>>> The question is: would anyone see using Cython as a problem?
>>>>>
>>>>> I know that the fewer dependencies, the better. However Cython is
>>>>> almost as ubiquitous as Scipy nowadays in Python's scientific
>>>>> stack.
>>>>>
>>>>> The only real drawbacks I can think of (and I can read about on the
>>>>> internet) are pain points on Windows and incompatibility with
>>>>> different
>>>>> Python implementations (e.g. PyPy). However, with pip wheels,
>>>>> Windows
>>>>> is a non problem (most Python distributions on Windows now also
>>>>> include
>>>>> Cython by default so manual compilation is far from the pain it
>>>>> used to
>>>>> be) and ASE/GPAW do not support PyPy anyway because of all the C
>>>>> dependencies.
>>>>>
>>>>> Note that I am *not* proposing to convert any existing code to
>>>>> Cython
>>>>> nor to use Cython by default. I am talking about specific cases
>>>>> which
>>>>> would currently be handled by writing pure C code.
>>>>>
>>>>> There are already a few cases where performance becomes an issue,
>>>>> so
>>>>> this is worth thinking about.  Can Cython be an optional
>>>>> dependency?
>>>>>
>>>>> Yes it can and at two different levels:
>>>>>
>>>>>      1) *Provide fallback functions* Nothing would really prevent us
>>>>> from
>>>>>      providing two different versions of hot functions and degrade
>>>>> to
>>>>>      pure Python ones upon import if needed. However this would
>>>>> require
>>>>>      to keep function APIs in sync in two different places.
>>>>>
>>>>>      On the bright side, the slow and hopefully correct Python
>>>>> function
>>>>>      could be used as a reference in unit tests: these would rarely
>>>>> be
>>>>>      changed and most of the optimisation would occur in the Cython
>>>>> ones
>>>>>      instead.
>>>>>
>>>>>      2) *Provide C/C++ files* Ultimately, Cython generates a bunch
>>>>> of C
>>>>>      files containing Cython-defined code + everything needed to
>>>>> create a
>>>>>      native Python module. Directly using those C files (thus not
>>>>>      requiring Cython itself) is just a matter of tracking them in
>>>>> git!
>>>>>      :) That kind of optional Cython support can be reflected in
>>>>> setup.py
>>>>>      directly (look for USE_CYTHON    ):
>>>>>
>>>>>
>>>>> https://gist.github.com/ctokheim/6c34dc1d672afca0676a#setuppy-and-d
>>>>> isutils
>>>>>
>>>>> (note however that a C compiler is still needed).
>>>>>
>>>>>
>>>>> I would like "pip install ase" to always just work, and for that I
>>>>> think we
>>>>> would need 1).  Alternatively, we should provide wheels for all
>>>>> architectures (manylinux1, win32, macosx, ...), but I don't think
>>>>> we have
>>>>> the resources to do that.
>>>>>
>>>>> PS: You can always start with the slow Python version.
>>>>>
>>>>> Jens Jørgen
>>>>>
>>>>>
>>>>> Regards,
>>>>> Gaël
>>>>>
>>>>> Best regards
>>>>> Ask
>>>>>
>>>>> Best regards,
>>>>> Gaël
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ase-users mailing list
>>>>> ase-users at listserv.fysik.dtu.dk
>>>>> https://listserv.fysik.dtu.dk/mailman/listinfo/ase-users
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ase-users mailing list
>>>>> ase-users at listserv.fysik.dtu.dk
>>>>> https://listserv.fysik.dtu.dk/mailman/listinfo/ase-users
>>>>>
>>>>>
>>
>> _______________________________________________
>> ase-users mailing list
>> ase-users at listserv.fysik.dtu.dk
>> https://listserv.fysik.dtu.dk/mailman/listinfo/ase-users
>
>
>
> _______________________________________________
> ase-users mailing list
> ase-users at listserv.fysik.dtu.dk
> https://listserv.fysik.dtu.dk/mailman/listinfo/ase-users