gensim: added Gensim 2.1.0 and related tools #25804

aptlin · 2017-05-15T14:34:30Z

Motivation for this change

Making NLP analysis easier with Nix.

Things done

Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- Linux
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

mention-bot · 2017-05-15T14:34:41Z

@sdll, thanks for your PR! By analyzing the history of the files in this pull request, we identified @zimbatm, @zraexy, @vcunat and @FRidh to be potential reviewers.

FRidh

Looks good. Some minor changes are needed. I haven't tested the PR though.

FRidh · 2017-05-15T14:49:35Z

pkgs/development/python-modules/gensim.nix

+    sha256 = "1wn7bji9b80wn1yggmh7a0dlwzdjr6cp24x4p33j2rf29lxnm2kc";
+  };
+
+  propagatedBuildInputs = [


indentation

FRidh · 2017-05-15T14:50:30Z

pkgs/development/python-modules/gensim.nix

+    morfessor
+  ];
+
+  checkPhase = ''


Note that currently this is the default checkPhase so its not necessary. No need to change it though.

FRidh · 2017-05-15T14:52:27Z

pkgs/development/python-modules/pyro4.nix

+    };
+
+    propagatedBuildInputs = [ six ];
+


meta.license?

FRidh · 2017-05-15T14:53:34Z

pkgs/development/python-modules/pyro4.nix

+, msgpack
+, six
+}:
+with rec{


use let ... in instead

FRidh · 2017-05-15T14:53:48Z

pkgs/development/python-modules/smart_open.nix

+, moto
+, responses
+}:
+with rec{


use let ... in instead

FRidh · 2017-05-15T14:53:59Z

pkgs/development/python-modules/smart_open.nix

+
+    # fails to find a test.support module
+    doCheck = false;
+  };


meta.license?

FRidh · 2017-05-15T14:54:39Z

pkgs/top-level/python-packages.nix

+
+  morfessor = callPackage ../development/python-modules/morfessor.nix { };
+
+  gensim = callPackage ../development/python-modules/gensim.nix { };  


even though the order of the attributes in this file is a mess, do try to stick to it.

@FRidh, do you want me to order them alphabetically? They are in the order of the commits. Unfortunately, I did not know all the missing dependencies at the beginning, so I propose to leave the order as it is now.

Try to find for each attribute an appropriate place in the quasi-alphabetically ordered attributes

What is the point, @FRidh, if they are all related to one package -- gensim? Spreading them over the entire file would only increase the entropy, it seems.

aptlin · 2017-05-15T15:33:27Z

Thank you very much for the review, @FRidh, I have updated the files with the changes requested.

FRidh · 2017-05-15T15:44:15Z

Oh, I now noticed one aspect that has to be fixed. Your expressions are in files named after the package. Instead, the rule is to put them in a folder, named after the package.

aptlin · 2017-05-15T16:04:10Z

But there are other packages, like numpy, which are not in their own folder -- should they also be, in principle, moved to their own folders?

FRidh · 2017-05-15T16:07:43Z

But there are other packages, like numpy, which are not in their own folder -- should they also be, in principle, moved to their own folders?

Yes, they should.

aptlin · 2017-05-15T16:33:32Z

@FRidh, I have moved all the recipes related to the PR.

aptlin · 2017-05-17T09:48:28Z

@FRidh, Travis CI on OS X seems to fail due to the following error:

pyemd/emd.cpp:458:10: fatal error: 'vector' file not found

Do you think we should try to fix it by specifying GCC_INSTALL_HOME?
See a similar question discussed here: harvard-acc/LLVM-Tracer#7

I have also updated the recipe for smart_open, which build failed because it's dependency, boto, is tailored to Python2 and decided to try to fix it by replacing boto with boto3.

I have also changed the doCheck boolean for selectors in the recipy for pyro4.

aptlin · 2017-05-17T11:27:12Z

@FRidh, the problem with importing urllib2 persists in the Travis CI build of boto for GNU/Linux (I am not sure why boto and boto3 are built). Shall I disable checking of boto in python3 with doCheck = isPy27; in a separate PR?

FRidh · 2017-05-17T11:41:23Z

@sdll just ignore Travis, we don't rely on it.

Currently I get

$ nix-build -A python.pkgs.gensim
...
/build/gensim-2.1.0/dist /build/gensim-2.1.0
Processing ./gensim-2.1.0-cp27-cp27mu-linux_x86_64.whl
Requirement already satisfied: scipy>=0.7.0 in /nix/store/xg1v4g2hzcvj7raakaf1gvxhsq3l95rb-python2.7-scipy-0.18.1/lib/python2.7/site-packages (from gensim==2.1.0)
Requirement already satisfied: six>=1.5.0 in /nix/store/m0apm5s8w43q8r9q9ciz3xj6vvxj5xci-python2.7-six-1.10.0/lib/python2.7/site-packages (from gensim==2.1.0)
Requirement already satisfied: smart-open>=1.2.1 in /nix/store/0l6jv4ip60gk8wvydsiki4a3ys09jr46-python2.7-smart_open-1.5.2/lib/python2.7/site-packages (from gensim==2.1.0)
Requirement already satisfied: numpy>=1.3 in /nix/store/2r7iirb4h95rnvvf4cns7471gkibnpsv-python2.7-numpy-1.11.3/lib/python2.7/site-packages (from gensim==2.1.0)
Collecting boto>=2.32 (from smart-open>=1.2.1->gensim==2.1.0)
  Could not find a version that satisfies the requirement boto>=2.32 (from smart-open>=1.2.1->gensim==2.1.0) (from versions: )
No matching distribution found for boto>=2.32 (from smart-open>=1.2.1->gensim==2.1.0)

builder for ‘/nix/store/6md6n0vqc9r58zfm8zq7p6d6am1349n0-python2.7-gensim-2.1.0.drv’ failed with exit code 1
error: build of ‘/nix/store/6md6n0vqc9r58zfm8zq7p6d6am1349n0-python2.7-gensim-2.1.0.drv’ failed

aptlin · 2017-05-17T11:53:50Z

@FRidh, so we need to use boto then, not boto3, sorry for the confusion -- I checked only if smart_open builds fine. I have reverted to boto. Could you please try again?

wldhx · 2017-06-07T22:32:44Z

@FRidh Is there anything preventing this from being merged? I just rebased this and built gensim locally with no issues.

jyp · 2017-06-22T09:19:44Z

I've done some overlapping changes in PR #26525. Perhaps you can rebase your work on master?

FRidh · 2017-08-13T20:14:46Z

Closing because #26525 was merged. If there are other changes you would like to get in, please open a new PR.

lib/maintainers.nix: add sdll

4cb5276

FRidh requested changes May 15, 2017

View reviewed changes

FRidh added 6.topic: python 8.has: package (new) This PR adds a new package labels May 15, 2017

aptlin force-pushed the gensim-2.1.0 branch from ce9a77f to 4382352 Compare May 15, 2017 15:32

aptlin force-pushed the gensim-2.1.0 branch from 4382352 to 9106102 Compare May 15, 2017 16:33

aptlin force-pushed the gensim-2.1.0 branch 2 times, most recently from 61aba52 to 277f1f5 Compare May 17, 2017 10:21

FRidh self-assigned this May 17, 2017

aptlin force-pushed the gensim-2.1.0 branch 2 times, most recently from eb9486f to 3b44a28 Compare May 17, 2017 11:07

aptlin mentioned this pull request May 17, 2017

spacy: added Explosion AI spaCy, thinc and related tools #25791

Closed

7 tasks

Sasha Delly added 6 commits May 17, 2017 11:52

smart_open: init at 1.5.2

3d77913

serpent: init at 1.19

8048379

pyro4: init at 4.59

5b5fffa

pyemd: init at 0.4.3

abd98f8

morfessor: init at 2.0.2a4

7592c43

gensim: init at 2.1.0

590b31e

aptlin force-pushed the gensim-2.1.0 branch from 3b44a28 to 590b31e Compare May 17, 2017 11:52

FRidh closed this Aug 13, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gensim: added Gensim 2.1.0 and related tools #25804

gensim: added Gensim 2.1.0 and related tools #25804

aptlin commented May 15, 2017

mention-bot commented May 15, 2017

FRidh left a comment

FRidh May 15, 2017

FRidh May 15, 2017

FRidh May 15, 2017

FRidh May 15, 2017

FRidh May 15, 2017

FRidh May 15, 2017

FRidh May 15, 2017

aptlin May 15, 2017 •

edited

Loading

FRidh May 15, 2017

aptlin May 15, 2017

aptlin commented May 15, 2017

FRidh commented May 15, 2017

aptlin commented May 15, 2017

FRidh commented May 15, 2017

aptlin commented May 15, 2017 •

edited

Loading

aptlin commented May 17, 2017 •

edited

Loading

aptlin commented May 17, 2017 •

edited

Loading

FRidh commented May 17, 2017

aptlin commented May 17, 2017 •

edited

Loading

wldhx commented Jun 7, 2017

jyp commented Jun 22, 2017

FRidh commented Aug 13, 2017


		morfessor = callPackage ../development/python-modules/morfessor.nix { };

		gensim = callPackage ../development/python-modules/gensim.nix { };

gensim: added Gensim 2.1.0 and related tools #25804

gensim: added Gensim 2.1.0 and related tools #25804

Conversation

aptlin commented May 15, 2017

Motivation for this change

Things done

mention-bot commented May 15, 2017

FRidh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aptlin May 15, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aptlin commented May 15, 2017

FRidh commented May 15, 2017

aptlin commented May 15, 2017

FRidh commented May 15, 2017

aptlin commented May 15, 2017 • edited Loading

aptlin commented May 17, 2017 • edited Loading

aptlin commented May 17, 2017 • edited Loading

FRidh commented May 17, 2017

aptlin commented May 17, 2017 • edited Loading

wldhx commented Jun 7, 2017

jyp commented Jun 22, 2017

FRidh commented Aug 13, 2017

aptlin May 15, 2017 •

edited

Loading

aptlin commented May 15, 2017 •

edited

Loading

aptlin commented May 17, 2017 •

edited

Loading

aptlin commented May 17, 2017 •

edited

Loading

aptlin commented May 17, 2017 •

edited

Loading