Fix dh-virtualenv packages build, avoid updating pip to latest v10.0

[arm4b]

Related to: StackStorm/st2-dockerfiles#58 StackStorm/st2-dockerfiles#59 StackStorm/st2-dockerfiles#60

pip recently released new v10.0 which broke the dh-virtualenv packages build.

So here is the rabbit hole.
While we use & pin virtualenv 15.1.0 which advertizes to install pip 9.0.1 (https://virtualenv.pypa.io/en/stable/changes/#id1) it doesn't happen in reality.

Obviously, at it happens in software world, it's not a bug but a feature, - virtualenv since 14.0.1 installs latest available (!) pip version by default when user creates a new virtualenv (see pypa/virtualenv#1044 and pypa/virtualenv#1157).

To avoid this behavior in virtualenv and stick with expected associated pip version, --no-download flag was added (see https://virtualenv.pypa.io/en/stable/changes/#id12):

Download new releases of the preinstalled software from PyPI when there are new releases available. This behavior can be disabled using --no-download.

Well, it means that once we updated to newer virtualenv version (14.0.1+) which installs latest pip by default, we opened a can of worms.

TODO

• Find the root cause
• Provide --no-download flag to virtualenv in st2-packages
  • Investigate new fails: we rely on our outdated fork of dh-virtualenv which doesn't have --extra-virtualenv-arg
    • Update our dh-virtualenv fork, re-implement custom changes StackStorm/dh-virtualenv@bf43b0a (thanks @Kami Add sendmail to the utuils used by the st2 linux pack st2-dockerfiles#61)
      • trusty build started to fail due to updated dh-virtualenv fork changes, fix: StackStorm/dh-virtualenv@bab5d2e
• Check if st2 repo needs --no-download as well for st2 pack install (thanks @Kami Pass --no-download flag to virtualenv binary when we create pack virtualenv st2#4085)

[bigmstone]

@armab I assume tests will be 💚once you update dh-virtualenv fork?

Just did a bit more digging into the circle report (Hate when I have to download the output). Answered my own question.

[Kami]

Good catch - I confirmed this nasty virtualenv env behavior locally.

vagrant@local$ virtualenv footest
New python executable in /data/st2docs/footest/bin/python
Installing setuptools, pip, wheel...done.
vagrant@local$ footest/bin/pip --version
pip 10.0.0 from /data/st2docs/footest/local/lib/python2.7/site-packages/pip (python 2.7)
vagrant@local$ virtualenv --no-download footest2
New python executable in /data/st2docs/footest2/bin/python
Installing setuptools, pip, wheel...done.
vagrant@local$ footest2/bin/pip --version
pip 9.0.1 from /data/st2docs/footest2/local/lib/python2.7/site-packages (python 2.7)

It looks like we might indeed need to update all of our Makefiles where we use virtualenv to create virtualenv and pass in --no-download to use pip 9.0.1 for the time being.

[Kami]

As far as st2 pack install goes - I confirmed it results in the same behavior since we also use latest version of virtualenv there.

So while it does install latest version of pip (10.0.0) it doesn't seem to break packs. In theory it shouldn't, because (in ideal world) packs should never import from pip or do anything like that.

Having said that, I still think it's probably safer and better (and results in more reproducible virtualenv creations) to pass --no-download flag to virtualenv binary while creating a pack virtual environment.

[Kami]

Nice work - thanks!

I will merge two st2 PRs into master, test them on staging unstable on all the distros (really just "st2 pack install" needs to be tested) and start working on 2.7.1 when everything is in order :)