Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Many links to packaging.python.org broken #1026

Closed
takluyver opened this issue Dec 6, 2021 · 26 comments
Closed

Many links to packaging.python.org broken #1026

takluyver opened this issue Dec 6, 2021 · 26 comments
Assignees
Labels
help wanted type: bug A confirmed bug or unintended behavior type: task Something that needs to be done that is not a bug or feature

Comments

@takluyver
Copy link
Member

I noticed a broken link from the Warehouse help pages to packaging.python.org, and was going to fix it, when I realised every link from Warehouse to packaging.python.org is broken. As are links here on the issue tracker, and links on Discourse posted up to yesterday.

Specifically, a URL like https://packaging.python.org/specifications/core-metadata/ needs an added /en/latest/ to work again. I'm guessing it used to have the 'single version' checkbox selected on RTD, and no longer does ("Use this with caution, only turn it on if you will never have multiple versions of your docs.")

IMO, we should either re-enable that setting ASAP (before people share too many links with the new scheme), or add redirects on RTD so existing links still work.

@bhrutledge
Copy link
Contributor

Thanks for pointing this out. Is there any indication of when this behavior changed? I don't have access to the RTD settings.

I wonder if it has to do with translations, since latest is still the only version listed:

Screen Shot 2021-12-06 at 8 52 03 AM

CC @webknjaz @meowmeowmeowcat

@takluyver
Copy link
Member Author

There were some links posted on Discourse at 16:02 UTC yesterday using the old scheme, so I'd guess it was still working then. I don't know of any way to see when exactly it was changed in RTD.

@bhrutledge
Copy link
Contributor

@webknjaz Could this be related to #1023?

@webknjaz
Copy link
Member

webknjaz commented Dec 6, 2021

This is a side-effect of fixing #927 (comment) (TL;DR translations don't work with a "single version mode" enabled). I don't have visibility into the main RTD setup for the English site but I suppose we could collect all the affected documents and manually add redirects for those. This would need to be done by @di or @ewdurbin. I can only provide the list of the URIs to map, it can be scraped via https://webknjaz.github.io/intersphinx-untangled/packaging.python.org/ or via intersphinx directly.

@webknjaz webknjaz added type: task Something that needs to be done that is not a bug or feature type: bug A confirmed bug or unintended behavior help wanted labels Dec 6, 2021
@webknjaz
Copy link
Member

webknjaz commented Dec 6, 2021

This should be the full list:

$ python -m sphinx.ext.intersphinx https://packaging.python.org/en/latest/objects.inv | awk -F': ' '{print$2}' | sed 's/#[^#]\+$//g' | sort -u

contribute/
discussions/
discussions/deploying-python-applications/
discussions/install-requires-vs-requirements/
discussions/pip-vs-easy-install/
discussions/wheel-vs-egg/
genindex/
glossary/
guides/
guides/analyzing-pypi-package-downloads/
guides/creating-and-discovering-plugins/
guides/distributing-packages-using-setuptools/
guides/dropping-older-python-versions/
guides/hosting-your-own-index/
guides/index-mirrors-and-caches/
guides/installing-scientific-packages/
guides/installing-stand-alone-command-line-tools/
guides/installing-using-linux-tools/
guides/installing-using-pip-and-virtual-environments/
guides/making-a-pypi-friendly-readme/
guides/migrating-to-pypi-org/
guides/multi-version-installs/
guides/packaging-binary-extensions/
guides/packaging-namespace-packages/
guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/
guides/single-sourcing-package-version/
guides/supporting-multiple-python-versions/
guides/supporting-windows-using-appveyor/
guides/tool-recommendations/
guides/using-manifest-in/
guides/using-testpypi/
key_projects/
news/
overview/
py-modindex/
search/
specifications/
specifications/binary-distribution-format/
specifications/core-metadata/
specifications/declaring-build-dependencies/
specifications/declaring-project-metadata/
specifications/dependency-specifiers/
specifications/direct-url/
specifications/entry-points/
specifications/platform-compatibility-tags/
specifications/pypirc/
specifications/recording-installed-packages/
specifications/simple-repository-api/
specifications/source-distribution-format/
specifications/version-specifiers/
support/
tutorials/
tutorials/creating-documentation/
tutorials/installing-packages/
tutorials/managing-dependencies/
tutorials/packaging-projects/

@takluyver
Copy link
Member Author

I think that if you add a 'prefix redirect' in RTD with no prefix, any URLs which would 404 would instead be redirected to the equivalent page with /en/latest/ prefixed. I haven't tried this, so take it with a grain of salt, but it might be easier than adding lots of separate redirects.

@webknjaz
Copy link
Member

webknjaz commented Dec 6, 2021

Yeah, I don't remember if it's possible to do for 404 only. Maybe listing just top-level routes would do it:

$ python -m sphinx.ext.intersphinx https://packaging.python.org/en/latest/objects.inv | awk -F': ' '{print$2}' | sed 's/#[^#]\+$//g;s#^\(\w\+/\).*#\1#g' | sort -u

contribute/
discussions/
genindex/
glossary/
guides/
key_projects/
news/
overview/
py-modindex/
search/
specifications/
support/
tutorials/

@webknjaz
Copy link
Member

webknjaz commented Dec 6, 2021

There's also https://sphinx-notfound-page.rtfd.io that can be used as a last resort.

@webknjaz
Copy link
Member

webknjaz commented Dec 6, 2021

Looks like Thomas is right and we should be able to get away with a prefix redirect per https://docs.readthedocs.io/en/stable/user-defined-redirects.html#prefix-redirects.

cc @di @ewdurbin

@takluyver
Copy link
Member Author

The RTD docs page seems pretty clear that it's 404s only. One bit does say 'for the time being', but if they change that I'd hope they do so in a way that requires opting in (e.g. a new option in the 'redirect types' list).

I don't think any of the current options allow mapping one prefix to a configurable different one (e.g. specifications to /en/latest/specifications). I think your only options here are stripping a configurable prefix and replacing it with /en/latest/, or setting up an 'exact redirect' for each individual page.

@ewdurbin
Copy link
Member

ewdurbin commented Dec 6, 2021

Added prefix redirect. Does that resolve the issue?

ee-imac:~ ee$ curl -s -I  https://packaging.python.org/contribute/ | grep location
location: https://packaging.python.org/en/latest/contribute/

@webknjaz
Copy link
Member

webknjaz commented Dec 6, 2021

I think this can be closed now but I'll leave it to @takluyver so that we have another confirmation.

@webknjaz webknjaz assigned takluyver and unassigned di and ewdurbin Dec 6, 2021
@bhrutledge
Copy link
Contributor

@bhrutledge bhrutledge reopened this Dec 7, 2021
@webknjaz
Copy link
Member

webknjaz commented Dec 7, 2021

However, I'm not sure what the right page is; maybe packaging.python.org/en/latest/tutorials/packaging-projects?

Looks like it should be https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/. The old tutorial has been converted into a guide in 2018: #498. You can see the file rename/move in the Files tab.

@bhrutledge
Copy link
Contributor

Hm. From a previously successful build of Twine's docs, it looks like it was redirecting to https://packaging.python.org/tutorials/packaging-projects/. Of course, Twine's docs can be updated, but I assume there's other use of the old URL on the Internet that is now broken. Can we restore the redirect (and others that might have been similarly lost)?

@webknjaz
Copy link
Member

webknjaz commented Dec 7, 2021

I don't have visibility into what's configured and I'm not sure if there's a way to see the previous configs. We may just have to live with it as it is now.

@bhrutledge
Copy link
Contributor

@ewdurbin Do you have any insight? Can you see if there are other redirects that were overridden/lost when the prefix redirect was added?

@ewdurbin
Copy link
Member

ewdurbin commented Dec 7, 2021

Below are all configured redirects, I'm guessing that the prefix redirect is clobbering some of the existing ones but haven't looked closely into how they interact.

Redirects table (outdated)
Type From URL To URL
prefix null null
page /distributing/ /tutorials/packaging-projects
page /distributing /tutorials/packaging-projects
page /tutorials/distributing-packages /tutorials/packaging-projects
exact /guides/installing-using-pip-and-virtualenv/ /guides/installing-using-pip-and-virtual-environments/
page /guides/installing-using-pip-and-virtualenv /guides/installing-using-pip-and-virtual-environments/
sphinx_htmldir null null
page /additional /guides
page /quickstart /tutorials
page /platforms /guides/installing-scientific-packages
page /tutorial /tutorials
page /installing /tutorials/installing-packages
page /deployment /discussions/deploying-python-applications
page /requirements /discussions/install-requires-vs-requirements
page /pip_easy_install /discussions/pip-vs-easy-install
page /wheel_egg /discussions/wheel-vs-egg
page /plugin_discovery /guides/creating-and-discovering-plugins
page /self_hosted_repository /guides/hosting-your-own-index
page /mirrors /guides/index-mirrors-and-caches
page /science /guides/installing-scientific-packages
page /install_requirements_linux /guides/installing-using-linux-tools
page /multi_version_install /guides/multi-version-installs
page /extensions /guides/packaging-binary-extensions
page /namespace_packages /guides/packaging-namespace-packages
page /single_source_version /guides/single-sourcing-package-version
page /multiple_python_versions /guides/supporting-multiple-python-versions
page /appveyor /guides/supporting-windows-using-appveyor
page /current /guides/tool-recommendations
page /installing /tutorials/installing-packages
page /installing/ /tutorials/installing-packages
page /additional/ /guides
page /quickstart/ /tutorials
page /platforms/ /guides/installing-scientific-packages
page /tutorial/ /tutorials
page /installing/ /tutorials/installing-packages
page /deployment/ /discussions/deploying-python-applications
page /requirements/ /discussions/install-requires-vs-requirements
page /pip_easy_install/ /discussions/pip-vs-easy-install
page /wheel_egg/ /discussions/wheel-vs-egg
page /plugin_discovery/ /guides/creating-and-discovering-plugins
page /self_hosted_repository/ /guides/creating-and-discovering-plugins
page /mirrors/ /guides/index-mirrors-and-caches
page /science/ /guides/installing-scientific-packages
page /install_requirements_linux/ /guides/installing-using-linux-tools
page /multi_version_install/ /guides/multi-version-installs
page /extensions/ /guides/packaging-binary-extensions
page /namespace_packages/ /guides/packaging-namespace-packages
page /single_source_version/ /guides/single-sourcing-package-version
page /multiple_python_versions/ /guides/supporting-multiple-python-versions
page /appveyor/ /guides/supporting-windows-using-appveyor
page /current/ /guides/tool-recommendations
page /current.html /guides/tool-recommendations
exact /en/latest/current/ /guides/tool-recommendations/
exact /en/latest/current /guides/tool-recommendations/
exact /tutorials/distributing-packages/ /en/latest/tutorials/packaging-projects

@dukecat0
Copy link
Member

dukecat0 commented Dec 7, 2021

@ewdurbin Is it possible to update the To URL section on the right-hand side? Probably adding en/latest to each of these URLs?

@bhrutledge
Copy link
Contributor

@meowmeowmeowcat I don't think that would resolve the issue that I reported above. I can see that there's a redirect from /tutorials/distributing-packages/ to /tutorials/packaging-projects. However, opening https://packaging.python.org/tutorials/distributing-packages/ instead redirects to https://packaging.python.org/en/latest/tutorials/distributing-packages/, which doesn't exist.

@takluyver
Copy link
Member Author

https://packaging.python.org/tutorials/distributing-packages/ works for me - I get redirected to https://packaging.python.org/en/latest/tutorials/packaging-projects/ . I don't know if something has changed, or if there's some randomness in the order that the redirects are processed.

@ewdurbin
Copy link
Member

ewdurbin commented Dec 7, 2021

doing a little testing on that specific URL now, which is why @takluyver got the expected result.

looks like if we change the "page" type redirects to "exact" we can restore these URLs, but I don't have time to script the specifics at this very moment.

@bhrutledge
Copy link
Contributor

Is there any more work planned for this issue? Maybe @ewdurbin?

@ewdurbin
Copy link
Member

All existing page redirects have been updated to "exact", the current redirect configuration is now as shown below.

Seems that caching might take some time to steady out... but this should resolve existing links being broken

Type From URL To URL
exact /tutorials/distributing-packages/ /tutorials/packaging-projects
exact /current.html /guides/tool-recommendations
exact /current/ /guides/tool-recommendations
exact /appveyor/ /guides/supporting-windows-using-appveyor
exact /multiple_python_versions/ /guides/supporting-multiple-python-versions
exact /single_source_version/ /guides/single-sourcing-package-version
exact /namespace_packages/ /guides/packaging-namespace-packages
exact /extensions/ /guides/packaging-binary-extensions
exact /multi_version_install/ /guides/multi-version-installs
exact /install_requirements_linux/ /guides/installing-using-linux-tools
exact /science/ /guides/installing-scientific-packages
exact /mirrors/ /guides/index-mirrors-and-caches
exact /self_hosted_repository/ /guides/creating-and-discovering-plugins
exact /plugin_discovery/ /guides/creating-and-discovering-plugins
exact /wheel_egg/ /discussions/wheel-vs-egg
exact /pip_easy_install/ /discussions/pip-vs-easy-install
exact /requirements/ /discussions/install-requires-vs-requirements
exact /deployment/ /discussions/deploying-python-applications
exact /installing/ /tutorials/installing-packages
exact /tutorial/ /tutorials
exact /platforms/ /guides/installing-scientific-packages
exact /quickstart/ /tutorials
exact /additional/ /guides
exact /installing/ /tutorials/installing-packages
exact /installing /tutorials/installing-packages
exact /current /guides/tool-recommendations
exact /appveyor /guides/supporting-windows-using-appveyor
exact /multiple_python_versions /guides/supporting-multiple-python-versions
exact /single_source_version /guides/single-sourcing-package-version
exact /namespace_packages /guides/packaging-namespace-packages
exact /extensions /guides/packaging-binary-extensions
exact /multi_version_install /guides/multi-version-installs
exact /install_requirements_linux /guides/installing-using-linux-tools
exact /science /guides/installing-scientific-packages
exact /mirrors /guides/index-mirrors-and-caches
exact /self_hosted_repository /guides/hosting-your-own-index
exact /plugin_discovery /guides/creating-and-discovering-plugins
exact /wheel_egg /discussions/wheel-vs-egg
exact /pip_easy_install /discussions/pip-vs-easy-install
exact /requirements /discussions/install-requires-vs-requirements
exact /deployment /discussions/deploying-python-applications
exact /installing /tutorials/installing-packages
exact /tutorial /tutorials
exact /platforms /guides/installing-scientific-packages
exact /quickstart /tutorials
exact /additional /guides
exact /guides/installing-using-pip-and-virtualenv /guides/installing-using-pip-and-virtual-environments/
exact /distributing /tutorials/packaging-projects
exact /distributing/ /tutorials/packaging-projects/
exact /tutorials/distributing-packages/ /en/latest/tutorials/packaging-projects
prefix null null
exact /guides/installing-using-pip-and-virtualenv/ /guides/installing-using-pip-and-virtual-environments/
sphinx_htmldir null null

@bhrutledge
Copy link
Contributor

@ewdurbin Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted type: bug A confirmed bug or unintended behavior type: task Something that needs to be done that is not a bug or feature
Projects
None yet
Development

No branches or pull requests

6 participants