spacy: added Explosion AI spaCy, thinc and related tools

[aptlin]

Motivation for this change

Making NLP analysis easier with Nix.

Things done

• Tested using sandboxing
  (nix.useSandbox on NixOS,
  or option build-use-sandbox in nix.conf
  on non-NixOS)
• Built on platform(s)
  • NixOS
  • macOS
  • Linux
• Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Fits CONTRIBUTING.md.

---

[mention-bot]

@sdll, thanks for your PR! By analyzing the history of the files in this pull request, we identified @zimbatm, @zraexy, @vcunat and @FRidh to be potential reviewers.

[FRidh]

• Looks good overall
• Remove meta.platforms
• Preferably use fetchPypi
• Do not commit whitespace changes in python-packages.nix

[FRidh]

don't set platforms in Python packages. The function already takes care of that.

[FRidh]

It needs pytest and nose, but uses unittest as runner? That's odd.

[FRidh]

fetchPypi

[FRidh]

This is not a runtime dependency, or is it?

[aptlin]

@FRidh, thank you very much for your review!

I have made requested changes and cleaned up the code, removing whitespace trimming in python-packages.nix.

With regards to murmurhash, cython is a build dependency, as you can see here.

[FRidh]

With regards to murmurhash, cython is a build dependency, as you can see here.

A requirements.txt file doesn't say much, because its not used when building a package with setuptools, unless its explicitly read and added to e.g. install_requires. Looking at the setup.py file it seems it is only required to run setup and perform the build.

If a package is only used during build and install time, then it should be added to buildInputs. If it is needed at runtime or when another package that depends on it is build, it should be added to propagatedBuildInputs.

[aptlin]

@FRidh, thank you for teaching me to look into the code closer!

I have changed ftfy to make Travis CI builds work for GNU/Linux and OS X, opening an issue about it in @LuminosoInsight's repo here.

I have also changed murmurhash build inputs as requested.

[aptlin]

@FRidh, I moved recipes to satisfy the 'package/default.nix' format, as well as replaced with rec {....} with let ... in.

OS X Travis seems to time out, could you please tell me how I can decrease the build time? My first idea was to set doCheck = false; in some recipes exclusively for OS X, is there a way to do it?

[FRidh]

@sdll you can ignore Travis. While it is sometimes useful, we cannot really rely on it.

[aptlin]

@FRidh, is there anything else I need to change? Is the tag "needs: clean-up" still relevant?

[aptlin]

@FRidh, I have updated the recipe for ftfy, changing its version to 4.4.3, the testing issue was fixed (see rspeer/python-ftfy#75)

I have also added pip as a dependency for spaCy, because its download function would not work without it, adding the request (explosion/spaCy#1063) to fix the code to account for the read-only status of the package home directory.

[aptlin]

@FRidh,

Travis CI on OS X seems to fail again due to the same error as in #25804:

pyemd/emd.cpp:458:10: fatal error: 'vector' file not found

Similarly, do you think we should try to fix it by specifying GCC_INSTALL_HOME?
See a similar question discussed here: harvard-acc/LLVM-Tracer#7

Is there a deeper problem with how clang c++ headers are treated in the OS X build of Nix? Shall I file an issue?

[FRidh]

@knedlsepp is this related to #27649? (I'm just diving through old PR's)

[knedlsepp]

@FRidh: Seems highly likely. I'll try to rework my PR asap, so we can merge.

[FRidh]

This needs a minor rebase. I successfully built the 2.7 version of spacy but the 3.6 hang during building.

[FRidh]

I've pushed this to master.

[JonathanReeve]

Great work! Any idea how to install the data using Nix? (E.g. en_core_web_lg?)