Compress HTTP with GZip where available #837
Conversation
The server behind GitHub URLs supports GZip compressed content encoding, which we can leverage to compress the YAML database behind rosdep substantially. This reduces the typical bandwidth usage of a call to `rosdep update` by 84%, from 410KiB to 64KiB. It doesn't appear to change the runtime performance on a moderately fast internet connection, but might be very helpful on less performant connections.
Codecov Report
@@ Coverage Diff @@
## master #837 +/- ##
=======================================
Coverage 74.96% 74.96%
=======================================
Files 42 43 +1
Lines 3311 3311
=======================================
Hits 2482 2482
Misses 829 829
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
I'm always caught off guard when compression has to be enabled and handled rather than being the default. This is an easy enough handler to add and will be a huge benefit, especially for rosdep users who are far from the GitHub network or on constrained connections.
We've avoided additional dependencies like requests in ros-infrastructure packages in order to keep the dependency tree tight but someday we might look at https://urllib3.readthedocs.io For now though I still don't think it's worth the overhead.
The server behind GitHub URLs supports GZip compressed content encoding, which we can leverage to compress the YAML database behind rosdep substantially.
This reduces the typical bandwidth usage of a call to
rosdep updateby 84%, from 410KiB to 64KiB. It doesn't appear to change the runtime performance on a moderately fast internet connection, but might be very helpful on less performant connections.Note that there are a lot of other improvements we could make here, such as streaming the content from the URL directly through the GZip decompression to the YAML parser and using a context to close the connection instead of explicitly calling
.close(). We should consider those improvements when we drop support for Python 2.