'pip install' on our MacOS python 3.5 CI is flaky

[njsmith]

Example:

https://ci.cryptography.io/blue/organizations/jenkins/python-trio%2Ftrio/detail/PR-503/12/pipeline/

Collecting pyOpenSSL (from -r test-requirements.txt (line 4))

  Could not find a version that satisfies the requirement pyOpenSSL (from -r test-requirements.txt (line 4)) (from versions: )

Example:

https://ci.cryptography.io/blue/organizations/jenkins/python-trio%2Ftrio/detail/PR-503/8/pipeline/

Collecting pytest>=3.3 (from -r test-requirements.txt (line 1))

  Could not find a version that satisfies the requirement pytest>=3.3 (from -r test-requirements.txt (line 1)) (from versions: )

This is the same kind of message that you get when using an old pip on an old MacOS, and it can't handle PyPI's new TLS requirements. And it's suspicious that it seems to happen on the Python 3.5 builder only (so far). However... in both of those runs you can see that the first thing we do is use get-pip.py to upgrade to pip 10.0.1. The log verifies it:

++ pip --version
pip 10.0.1 from /Users/jenkins/workspace/-503-TALZSOV5BV3A6O3/trio/.venv/lib/python3.5/site-packages/pip (python 3.5)

Which makes sense, because if we didn't have working TLS, then we would be 100% unsuccessful at downloading things. Here we're only... intermittently, apparently at random, unsuccessful at downloading things.

I guess the first thing to do is to add some -vvv switches to the pip calls and wait for it to happen again...

[dstufft]

That was only the behavior of pip during the brownouts, since we're at 100% unavailability, we were able to have Fastly hard fail old TLS versions, and they give a OpenSSL error message now instead of just not being able to find versions.

[njsmith]

@dstufft Thanks for the information. That's mysterious, though... hopefully it'll happen again soon and we'll get more info. In the mean time, should I be filing a bug on pip to give better error messages when there's some kind of communication failure? That we're getting this weird "(from versions: )" message from two different issues is unfortunate...

[njsmith]

Okay, it happened again, this time with pip -vvv: https://ci.cryptography.io/blue/organizations/jenkins/python-trio%2Ftrio/detail/PR-512/1/pipeline

Relevant logs:

Collecting idna (from trio==0.4.0+dev)
  1 location(s) to search for versions of idna:
  * https://pypi.org/simple/idna/
  Getting page https://pypi.org/simple/idna/
  Looking up "https://pypi.org/simple/idna/" in the cache
  Current age based on date: 31182
  Freshness lifetime from max-age: 600
  Freshness lifetime from request max-age: 600
  https://pypi.org:443 "GET /simple/idna/ HTTP/1.1" 304 0
  Skipping page https://pypi.org/simple/idna/ because of Content-Type: unknown
  Could not find a version that satisfies the requirement idna (from trio==0.4.0+dev) (from versions: )

I'm not sure what that means but it sure looks weird.

[njsmith]

There seems to be evidence accumulating at pypa/pip#5345 that this is triggered by multiple pip install processes accessing the same cache at the same time. That suggests that we could use something like this as a workaround: pypa/pip#5345 (comment)

[njsmith]

Hopefully #519 worked around this for now, but we'll have to wait and see.

[njsmith]

I guess since the real bug is being tracked upstream, we can close those while we are waiting and seeing, and re-open it if the problem re-occurs.

[njsmith]

Ughhh this just showed up again: https://ci.cryptography.io/blue/organizations/jenkins/python-trio%2Ftrio/detail/PR-525/1/pipeline

That build definitely had the XDG_CACHE_HOME workaround applied:

+ export XDG_CACHE_HOME=/Users/jenkins/workspace/-525-PLUQ2ZA3A6CCQS3/pip-cache
+ XDG_CACHE_HOME=/Users/jenkins/workspace/-525-PLUQ2ZA3A6CCQS3/pip-cache
+ echo XDG_CACHE_HOME=/Users/jenkins/workspace/-525-PLUQ2ZA3A6CCQS3/pip-cache
XDG_CACHE_HOME=/Users/jenkins/workspace/-525-PLUQ2ZA3A6CCQS3/pip-cache

So apparently the workaround is not reliable! wtf. I'll post on the pip issue as well.

[njsmith]

Actually... there's something funky going on with the workaround. I thought that should be setting XDG_CACHE_HOME to a directory inside the jenkins workspace that gets cleared after each run, which would mean that the cache should always start out empty on each run. But in fact pip is finding things in the cache. So one of my assumptions is wrong.

[njsmith]

Oh, here's an idea... maybe pip doesn't actually respect XDG_CACHE_HOME on MacOS, given that it's... MacOS, and XDG_CACHE_HOME is a Linux thing.