rosdep cannot fetch https custom rules

[markpaters]

Setup: Ubuntu 16.04 LTS, rosdep installed via apt (but fails the same way if installed from source)

For the past couple of months, we've had a problem where rosdep cannot download a publicly accessible rules file from our gitlab site. The file can be downloaded using curl with no problem. Patching rosdep to use urllib3 rather than urllib2 fixes the problem (I can make this a pull request if you want to see it). I have some suspicion that this is due to gitlab's retirement of support for older TLS versions, but I have not been successful getting urllib2 to work in rosdep, even when forcing the protocol to TLS1.2, which is what curl uses when it succeeds.

Sample rules file URL for testing: https://gitlab.com/markpaters/test/raw/master/rosdep_test.yaml
Outcome with stock rosdep:

 $ rosdep update
reading in sources list data from /etc/ros/rosdep/sources.list.d
ERROR: unable to process source [https://gitlab.com/markpaters/test/raw/master/rosdep_test.yaml]:
	HTTP Error 403: Forbidden (https://gitlab.com/markpaters/test/raw/master/rosdep_test.yaml)

[christophebedard]

I have some suspicion that this is due to gitlab's retirement of support for older TLS versions, but I have not been successful getting urllib2 to work in rosdep, even when forcing the protocol to TLS1.2, which is what curl uses when it succeeds.

I've had a similar problem before with gitlab.com too: dirk-thomas/vcstool#155 (comment) It seems like it blocks certain user agents, so the fix might be as simple as setting the user agent to a non-default value, e.g. "rosdep/<version>".

[markpaters]

Thanks, @christophebedard ! This appears to be exactly what's going on. I've created a pull request along the lines of your logic referenced above.