gzip and nginx optimize module for static assets

[CrazyPython]

I'm asking because I want to submit a pull request letting static assets be compressed with gzip.

[agjohnson]

Our operational provisioning isn't part of this repository. We're happy to have any suggestions, but the provisioning and other operational pieces are all private projects.

[CrazyPython]

Can it be made open-source? There is a campaign going on to open-source not just the code but all the things used to maintain it. As of now, readthedocs.io is quite slow and I want to submit a PR that does some major performance tweaks.

[ericholscher]

What in particular are you wanting to have gzipped? We already have this config:

    gzip on;
    gzip_vary on;
    gzip_types    text/plain text/html text/css
                  application/x-javascript text/xml
                  application/xml application/xml+rss
                  text/javascript;

[ericholscher]

I've updated it in a PR to:

    gzip_types
    application/atom+xml application/javascript application/json application/ld+json application/manifest+json
    application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf
    application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp
    image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc
    text/vtt text/x-component text/x-cross-domain-policy;

[CrazyPython]

*.js ≈ 269KB

https://kintojs.readthedocs.io/en/latest/js/highlight.pack.js   183KB
https://kintojs.readthedocs.io/en/latest/js/jquery-2.1.1.min.js 53 KB

• cache *.js *.css

google has a NGINX module that does all of this automatically https://www.modpagespeed.com/doc/build_ngx_pagespeed_from_source

https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Fkintojs.readthedocs.io&tab=desktop

[ericholscher]

I'm not following. Are you wanting us to minify the JS along with gzipping it? I don't feel comfortable changing the actual served JS content of folks, other than compressing it, at scale.

[CrazyPython]

Google PageSpeed insights suggests:

• Enabling compression
• Eliminating render-blocking JavaScript and CSS in above-the-fold content
• Leveraging browser caching
• Minifying CSS and JS

And Google provides NGINX and Apache modules to apply those optimizations automatically

[humitos]

@CrazyPython I have just some comments/questions on this:

Enabling compression

This is already done. Just #3564 missing

Eliminating render-blocking JavaScript and CSS in above-the-fold content

What's needed for this? I think with the public code you can propose a PR, right? How complicated it is? How much work does this involve?

Leveraging browser caching

What is needed for this?

Minifying CSS and JS

From Eric's reply I think this is not going to be done in the near future.

[CrazyPython]

Would you consider using the NGINX module google made that does the optimizations automatically?

[davidfischer]

I largely agree with most of this. However some of it may go into other places.

Eliminating render-blocking JavaScript and CSS in above-the-fold content

I don't think RTD should modify the HTML output of docs builds to this extent. If an author has put inline JS into their docs (using a retructuredtext raw block) it might depend on loading libraries before executing that inline JS. Moving JS and if possible CSS down would improve performance, but I think individual Sphinx themes should do that, not RTD. In fact, there were bugs around this in the RTD theme. See readthedocs/sphinx_rtd_theme#545.

Minifying CSS and JS

Similar to the above point, this should happen but I would argue the theme should do it. I don't think RTD should be modifying people's build output in that way.

Leveraging browser caching

This is something that RTD can do, although I think it might be a bit more complex than just adding an nginx module. Of these 3 things, this will also make the largest performance impact for visitors on repeat visits to RTD.

Take for example, the pageload for https://docs.readthedocs.io/en/latest/ :

• Loads 2 CSS files from media.readthedocs.org and 1 (typically a 404) from the build output
• Loads 7 font files all from media.readthedocs.org
• Loads 5 JS files from media.readthedocs.org and 3 from the build output

The default browser caching on media.readthedocs.org caches for 7 days and it uses eTags. There is no caching on the generated build output although there are eTags. The generated build output could change build to build (for example, some custom CSS a user has for overriding a theme) so I don't think RTD can do much there in terms of caching other than eTags which are already on. The files on media.readthedocs.org in theory can change on a new deploy of RTD although some are static forever in practice. This is where the opportunity for caching is, IMO.

One of the easiest optimizations would be to set multi-year/forever cache times for files we know never change. RTD could do this for the fonts, for doctools.js, underscore.js, jquery-2.0.3.min.js, and for jquery-migrate-1.2.1.min.js. readthedocs-doc-embed.css and readthedocs-doc-embed.js will need shorter cache times as those are subject to change. This could be done one-off in the nginx configs although it would be a bit ugly. This would reduce the necessary HTTP requests on repeat visits from 18 to 5. From a size perspective, the fonts are by far the largest files and they would be cached forever leading to significant improvement.

Since these files are written by ./manage.py collectstatic however, a smarter way might be to leverage Django's ManifestStaticFileStorage. Perhaps there's a way to read staticfiles.json and replace the the filename with a version that is cached forever. For example, rather than loading underscore.js we would load underscore.13ac50b.js which is cached forever and never changes unless the actual file contents change. This could possibly happen at docs build time or when the files are served (using ngx_http_sub_module maybe?).

[humitos]

Is there anything else actionable here that we want to do?

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further. Thanks!