scipy version

[marlinarnz]

In the requirements files, the scipy version is set to 1.6.1, but the scipy.sparse.coo_array, which is required from networkx in the step_road_pathfinder function, appears only in version 1.8.0 of scipy.

In general, it is rather unstable, to set specific versions in requirements files, if they are that long. You might consider setting minimum requirements instead. My quetzal virtual environment works best with updated libraries, which has the side-benefit of using new features like pd.merge(how='cross'). However, I guess this would require some environment testing on different machines.

[qchasserieau]

Hi Marlin,

Thank you for your feedback on the never-ending conundrum of the requirements.

We sometime set specific versions of the libraries in order to prevent compatibility issues that sometime rise between the libraries (most of the time, if it is set to a specific version in the requirements, we have experienced such lack of retro compatibility for the library).

However, as you said, the library versions of the requirements should be compatible for it is the very aim of the frozen versions in the first place. I must admit that we focused on the pip_requirements that are used for the recommended windows procedure (with wheels). Thank you for mentioning the incompatibility : where can we find you updated requirements file ?

What you preach is a flexible install procedure that uses the most recent libraries that appear to be compatible. It should be an option for experienced developers like you who want to benefit from the latest developments of their favorite libraries such as pd.merge(how='cross') . It is a dev install procedure : maybe your requirement file should be identified as so ?

What is also needed is a stable install procedure which works regardless of evolution of the syntax of quetzal's dependencies. A model that was built with quetzal 4 years ago is likely to use pandas' syntaxes in an outdated way (pandas behavior does change), therefore, it is necessary that Quetzal's version from 4 years ago installs pandas' version from 4 years ago. If you use today's pandas sources with pandas' 4 years old pandas syntax, it will not yield the same results.

Hence the two options we want to offer in this situation : either you install the latest dependencies with the dev install procedure and you update your model's syntax to be consistent. Or you run the stable install procedure and carry on with your outdated yet functioning model.

Let me know how you feel about that.

Cheers,
Quentin

[marlinarnz]

Hi Quentin,

that sounds like a good solution to me! In fact, I don't use pip for installing libraries (I find it troublesome), but conda. As conda does not accept requirements files, I did it manually and let conda choose the compatible versions. Hence, I don't have a requirements file, but the installed libraries are the same: The environment needs at least geopandas osmnx jupyter numba ray geopy pytables contextly gtfs-kit xlrd tqdm scipy. The installation works well from conda-forge and with all versions of Python >=3.8.

Do you think one should write a bat file (I don't have experience there)? Otherwise it might be a more flexible bugfix, if this hint was in the readme. Though, I guess it makes sense if one of you tried it first on your machines.

Cheers
Marlin