[ase-users] Opinion about potentially adding Cython as a dependency

[Jussi Enkovaara]

Hi,
generally, I think that Cython is nowadays quite useful for speeding up 
Python code, and if starting a new project from scratch I might even 
consider using Cython for some computational heavy kernels. However, I 
just want to point out that allowing ASE code to depend on Cython is 
pretty much equivalent of allowing C extensions in ASE. I guess having 
Cython as optional dependency might be fine, but as said, one should 
then accept optionally also C-code within ASE.

Best regards,
Jussi

On 2017-10-17 10:22, Jens Jørgen Mortensen via ase-users wrote:
> On 10/12/2017 04:01 PM, Gaël Donval wrote:
>>> If it can be done on a wholly opt-in basis then I think it is great.
>>> The question of C extensions and dependencies on compiled libraries
>>> have come up several times because they have legitimate applications,
>>> but anything that makes installation trickier will also affect users
>>> who don't need those features.  And then there's the question of
>>> release testing (platform dependence).
>> I suspect releasing wheels will solve the problem for most of us
>> anyway. Apparently, people usually take advantage CI systems to
>> generate theirs but I have not tried it myself.
>>
>> So my take out of this it: it is fine to use Cython provided that its
>> absence (or the absence of working compiler) doesn't affect people who
>> are not using the functionality. So I guess, at best, fall-back
>> function should be provided or, if really impractical, no failure
>> should occur as long as the specific functionalities are not used.
>>
>>> Can we establish a working minimal example where ASE builds and uses
>>> Cython to perform some trivial action, and see how it affects the
>>> build requiements?
>> I don't have time to do that just now but I can implement existing
>> functions in, say, ase/geometry/distance.py in cython this week-end. It
>> would be small and decoupled enough not to be too invasive and yet
>> provide a useful sandbox.
> 
> My preference would be to write C-code which most people can 
> read/understand/maintain and then write the interface to Python using 
> cython/cffi/ctypes.  And only do this if a Python+NumPy+SciPy solution 
> is 10 times slower or uses 10 times more memory.
> 
> Jens Jørgen
> 
>> Cheers,
>> Gaël
>>
>>
>>> Best regards
>>> Ask
>>>
>>> 2017-10-11 6:47 GMT+02:00 Jens Jørgen Mortensen <jjmo at dtu.dk>:
>>>> Den 10-10-2017 kl. 16:32 skrev Gaël Donval via ase-users:
>>>>
>>>> Dear Ask,
>>>>
>>>> Dear Gaël,
>>>>
>>>> 2017-10-10 12:52 GMT+02:00 Gaël Donval via ase-users
>>>> <ase-users at listserv.fysik.dtu.dk>:
>>>>
>>>> Dear list,
>>>>
>>>> I am currently working on some geometrical code that I will
>>>> ultimately
>>>> propose adding to ASE. I am currently using Cython to speed up what
>>>> needs speeding up and, to a more limited extent, I also use it to
>>>> very
>>>> conveniently generate bindings to C.
>>>>
>>>> The question is: would anyone see using Cython as a problem?
>>>>
>>>> I know that the fewer dependencies, the better. However Cython is
>>>> almost as ubiquitous as Scipy nowadays in Python's scientific
>>>> stack.
>>>>
>>>> The only real drawbacks I can think of (and I can read about on the
>>>> internet) are pain points on Windows and incompatibility with
>>>> different
>>>> Python implementations (e.g. PyPy). However, with pip wheels,
>>>> Windows
>>>> is a non problem (most Python distributions on Windows now also
>>>> include
>>>> Cython by default so manual compilation is far from the pain it
>>>> used to
>>>> be) and ASE/GPAW do not support PyPy anyway because of all the C
>>>> dependencies.
>>>>
>>>> Note that I am *not* proposing to convert any existing code to
>>>> Cython
>>>> nor to use Cython by default. I am talking about specific cases
>>>> which
>>>> would currently be handled by writing pure C code.
>>>>
>>>> There are already a few cases where performance becomes an issue,
>>>> so
>>>> this is worth thinking about.  Can Cython be an optional
>>>> dependency?
>>>>
>>>> Yes it can and at two different levels:
>>>>
>>>>      1) *Provide fallback functions* Nothing would really prevent us
>>>> from
>>>>      providing two different versions of hot functions and degrade
>>>> to
>>>>      pure Python ones upon import if needed. However this would
>>>> require
>>>>      to keep function APIs in sync in two different places.
>>>>
>>>>      On the bright side, the slow and hopefully correct Python
>>>> function
>>>>      could be used as a reference in unit tests: these would rarely
>>>> be
>>>>      changed and most of the optimisation would occur in the Cython
>>>> ones
>>>>      instead.
>>>>
>>>>      2) *Provide C/C++ files* Ultimately, Cython generates a bunch
>>>> of C
>>>>      files containing Cython-defined code + everything needed to
>>>> create a
>>>>      native Python module. Directly using those C files (thus not
>>>>      requiring Cython itself) is just a matter of tracking them in
>>>> git!
>>>>      :) That kind of optional Cython support can be reflected in
>>>> setup.py
>>>>      directly (look for USE_CYTHON    ):
>>>>
>>>>
>>>> https://gist.github.com/ctokheim/6c34dc1d672afca0676a#setuppy-and-d
>>>> isutils
>>>>
>>>> (note however that a C compiler is still needed).
>>>>
>>>>
>>>> I would like "pip install ase" to always just work, and for that I
>>>> think we
>>>> would need 1).  Alternatively, we should provide wheels for all
>>>> architectures (manylinux1, win32, macosx, ...), but I don't think
>>>> we have
>>>> the resources to do that.
>>>>
>>>> PS: You can always start with the slow Python version.
>>>>
>>>> Jens Jørgen
>>>>
>>>>
>>>> Regards,
>>>> Gaël
>>>>
>>>> Best regards
>>>> Ask
>>>>
>>>> Best regards,
>>>> Gaël
>>>>
>>>>
>>>> _______________________________________________
>>>> ase-users mailing list
>>>> ase-users at listserv.fysik.dtu.dk
>>>> https://listserv.fysik.dtu.dk/mailman/listinfo/ase-users
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> ase-users mailing list
>>>> ase-users at listserv.fysik.dtu.dk
>>>> https://listserv.fysik.dtu.dk/mailman/listinfo/ase-users
>>>>
>>>>
> 
> _______________________________________________
> ase-users mailing list
> ase-users at listserv.fysik.dtu.dk
> https://listserv.fysik.dtu.dk/mailman/listinfo/ase-users