Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add sticky caching for release descriptions #3745

Closed
wants to merge 1 commit into from

Conversation

theacodes
Copy link
Contributor

@theacodes theacodes commented Apr 18, 2018

This involves quite a bit of changes:

  • Two new fields on the Release model, rendered_description and renderer_version.
  • A new property (not field) on the Release model, html_description, which renders and caches the release description.
  • Removal of the readme filter.
  • New camoify filter separated from the former readme filter.
  • New warehouse.utils.readme separated from the former readme filter.

I am not totally convinced this is the best approach - I am more than open to moving things around. Reviewers, feel free to send commits to this branch.

Partially resolves #3739.

This involves quite a bit of changes:
* Two new fields on the Release model, `rendered_description` and `renderer_version`.
* A new property (not field) on the Releaes model, `html_description`, which renders and caches the release description.
* Removal of the `readme` filter.
* New `camoify` filter separated from the former `readme` filter.
* New `warehouse.utils.readme` separated from the former `readme` filter.

I am not totally convinced this is the best approach - I am more than open to moving things around. Reviewers, feel free to send commits to this branch.
@dstufft
Copy link
Member

dstufft commented Apr 18, 2018

I don't really like the "check for cache, serve or render inline and cache" approach here, because AIUI part of the problem here is that rendering is taking a long time, so while the cache solves it most of the time, it doesn't solve it for that initial page load.

Unfortunately, that leaves us in a bit of a sticky position, because of the initial render. For subsequent re-renders we can pretty easily serve the existing rendered value under the re-render has been successful, but the initial render you really only have a few choices:

  1. Do the initial render inline as part of the upload.
  2. Kick off an asynchronous task to render, and if someone loads the page in the meantime do something hinky to display a spinner and load the rendered description after the page load is over.
  3. Just deal with the fact if someone loads a page quickly enough after a release, that they're going to get possibly slow renders.

Of these, I think I prefer (1). It might make the upload route slower, but it's already pretty slow so that's not really much of a regression. It also has the nice benefit of allowing us to reject the upload if the passed in value doesn't render successfully. That's not a huge deal for markdown, since everything is valid Markdown, but it's a bigger deal for people using rst.

@ewdurbin
Copy link
Member

We can't do it until pypi-legacy is shutdown, but we could also take this opportunity to break long_descriptions and their rendered html into a separate model.

Release.long_description is a brute and dramatically complicates access to the Release table.

@dstufft
Copy link
Member

dstufft commented Apr 18, 2018

Release.long_description is a brute and dramatically complicates access to the Release table.

Does it really? To be clear, I'm not against it, but now that the Release.description field is deferred by default, it seems like typical use of the Release model is generally unaffected by the existence of that attribute?

@theacodes
Copy link
Contributor Author

theacodes commented Apr 18, 2018

So yeah, @dstufft has most of the same reservations I had about this: it's not ideal, but it's what we could do if we wanted to do this today. I'm totally fine to wait until legacy is shutdown and go with (1).

The great news is that a lot of the changes in this PR can be done ahead of time (such as splitting out the rendering logic and camoify). I'll send PRs to do that and close this one for now.

Let's revisit in May (maybe during PyCon?) and close this loop.

@theacodes theacodes closed this Apr 18, 2018
@dstufft
Copy link
Member

dstufft commented Apr 18, 2018

We only need to wait for legacy if we want to move Release.description to a different table, everything else (adding columns, etc) can happen now.

@theacodes
Copy link
Contributor Author

Oh, right, because uploads don't go through legacy anymore. Fair enough. Still going to split this into a few PRs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

cache/stash long_descriptions rendered by readme_render
3 participants