Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use the latest Sphinx version #125722

Open
AA-Turner opened this issue Oct 19, 2024 · 12 comments
Open

Use the latest Sphinx version #125722

AA-Turner opened this issue Oct 19, 2024 · 12 comments
Labels
docs Documentation in the Doc dir

Comments

@AA-Turner
Copy link
Member

AA-Turner commented Oct 19, 2024

Documentation

Python's documentation uses the Sphinx toolchain. Originally written for the Python documentation, it is now developed and distributed as an independent project. The two remain closely linked, e.g. when new syntax is added to the Python language. To document this syntax in the best way, Sphinx must support it and Python must require a version of Sphinx that supports the syntax.

Using the new syntax in PEP-695 as an example:

  1. The implementation was committed on 16 May (gh-103763: Implement PEP 695 #103764)
  2. The first draft of the documentation was written on 19 May (gh-103921: Document PEP 695 #104642)
  3. A feature request was created in Sphinx on 23 May (Support PEP 695 syntax for class definitions sphinx-doc/sphinx#11438)
  4. Sphinx added support on 23 July (Support PEP 695 and PEP 696 syntax in py:function and py:class directives. sphinx-doc/sphinx#11444)
  5. Sphinx published a release with support on 24 July (v7.1.0)
  6. The Python project can use those features, 19 October [+ 1 year] (GH-125277: Increase minimum supported Sphinx to 7.2.6 #125368)

This is a ~two month window from implementation to support in a release of Sphinx. It took a further 15 months and two feature releases (3.12, 3.13) until these features can be used in Python. Due to this, our documentation is meaningfully worse for readers and programmers. Using older versions of Sphinx mean that we cannot succinctly cover the features and syntax that exist within released versions of Python.

Core developers responsible for these features have expressed interest, but have been hampered by our self-imposed restriction of the minimum version of Sphinx that we support.

We adopt these restrictions for the benefit of downstream Linux redistributors, as can be seen in the more recent issues in the summary table below. This has not always been the case. From the adoption of Sphinx in 2007 (8ec7f65 / 116aa62) until 2014 (f7b2f36), the latest source-tree checkout of Sphinx at https://svn.python.org/ was used. From then until ~2018 the minimum required Sphinx version (controlled by needs_sphinx in conf.py) tracked the latest release promptly. This has since ossified.

In a recent informal discussion, several committers supported the idea of removing or relaxing this restriction, allowing the Python documentation to use the latest version(s) of Sphinx. This is the context for this note, which for now is a proposal. The status quo will prevail if there is not sufficient support.

As a concrete suggestion, I propose that when evaulating increasing the minimum required version of Sphinx, we no longer consult downstream redistributors' Sphinx versions. The procedure will follow standard Python development processes in all other ways. I expect that the minimum version would be updated if and when a new Sphinx feature is of sufficient benefit to Python. This, though, may be a greater version than a downstream redistributor provides.

We would like to solicit views from representatives of downstream redistributors as to how (in)feasible this proposal would be. My understanding is that Fedora and SUSE have processes whereby a newer version of Sphinx can be used soley for the Python documentation. I do not know how this will impact Debian or Gentoo.

Thank you in advance for your consideration,
Adam

Table of past changes to needs_sphinx

needs_sphinx Issue PR / Commit Latest Sphinx 1
7.2.6 #125277 #125368 8.1.3
6.2.1 #117928 #121986 7.4.6
4.2 #109209 #109210 7.2.6
3.2 #86986 #93337 5.0.1
1.8 #80188 #11887 1.8.4
1.7 doc-sig #9423 1.8.1
1.2 #65630 90d76ca 1.2.3
1.1 #64860 f7b2f36 1.2.1

Previous discussion relating to the minumum version:

cc:

Linked PRs

Footnotes

  1. At commit date.

@AA-Turner AA-Turner added the docs Documentation in the Doc dir label Oct 19, 2024
@willingc
Copy link
Contributor

@AA-Turner It would be helpful to have the latest version of Sphinx build the latest release of CPython. I suspect most users will view documentation on python.org.

@AA-Turner
Copy link
Member Author

Yep, and we currently use the latest version of Sphinx on docs.python.org (see the footer for "Created using ..."). This proposal would in effect mean we can narrow the range between the minimum supported and what we use in Doc/requirements.txt.

A

@hroncok
Copy link
Contributor

hroncok commented Oct 21, 2024

From the Fedora point of view, we have no trouble allowing the latest/development Python 3.X version documentation to use the latest version of Sphinx. My understanding it that this is usually the only Python version that needs to use the new features (such as new Python syntax). Once the features of a Python branch are stable, the required Sphinx version IMHO should also remain stable.

In the past, you proposed bumping the minimum Sphinx version for older Python versions, such as 3.12. That might sometimes be problematic for us. That said, your communication wrt. such bumps was always appreciated but never taken for granted. If you decide to always use the latest Sphinx for Python 3.12, 3.12, etc. at this point, it might be a bit challenging for us, but it won't be a blocker.

@AA-Turner
Copy link
Member Author

Thank you @hroncok, this is good to know.

In the past, you proposed bumping the minimum Sphinx version for older Python versions, such as 3.12. That might sometimes be problematic for us. [...] If you decide to always use the latest Sphinx for Python 3.12, 3.12, etc. at this point, it might be a bit challenging for us, but it won't be a blocker.

Our development workflow means that we backport almost all documentation PRs to stable versions of Python (currently 3.12 & 3.13). This is mainly so that the website at https://docs.python.org/3/ shows the latest documentation, as /3/ is a symlink to the most recent stable (currently /3.13).

This would mean that we would increase the minimum version for both HEAD and also all bugfix branches (or at least the most recent bugfix branch). The proposal wouldn't have much value without this, as it would effectively prohibit us from being able to confidently backport PRs.

Hopefully this is a little clearer. I think that it falls into the 'challenging but manageable' category of your response, if I'm understanding correctly.

A

@AA-Turner
Copy link
Member Author

@doko42 @mitya57 @mgorny would this proposal cause significant issues for Debian or Gentoo?

A

@mitya57
Copy link
Contributor

mitya57 commented Jan 11, 2025

It can cause some issues.

When a new major Sphinx version is released, it usually breaks a lot of reverse dependencies, in particular third-party extensions and themes. So I cannot upload it to Debian immediately, instead I need to wait a few months to give upstreams time to fix these issues. In many cases I have to submit my own PRs to various projects to support new Sphinx versions, to make them build fine in Debian.

For example, here is a list of packages which were broken by Sphinx 7 release. The test rebuild was done after Sphinx 7.1 was released, so quite some time has already passed since 7.0:
https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=sphinx7.1;[email protected]

To give another example: Sphinx 8.0 was released on Jul 29, 2024, but sphinx-rtd-theme (which is one of the most popular Sphinx themes) made a release with Sphinx 8.0 support only on Oct 7, 2024.

So if a new Python version requires a Sphinx version which was released a month or two earlier, it will be a problem. If more time has passed (like 4-6 months), then it is probably okay.

@AA-Turner
Copy link
Member Author

Thanks @mitya57. If we adopted this, it would mean using the latest Sphinx version for all stable Python versions (e.g. currently 3.12 and 3.13, and git HEAD (to-become 3.14)). I don't know how up-to-date Debian keeps the system Python (e.g. stable has 3.11), as Python 3.12 will fall into security-only in April 2025 (3.13: Oct 2026), meaning it would be frozen in terms of Sphinx.

Would this sort of timeline be likely to affect Debian? We could adopt a policy of 3 <= N <= 6 months, but that doesn't really achieve the goals cited in the initial post.

A

@mgorny
Copy link
Contributor

mgorny commented Jan 12, 2025

@doko42 @mitya57 @mgorny would this proposal cause significant issues for Debian or Gentoo?

Not for Gentoo, since we use prebuilt docs right now (and in general try not to stay behind with Sphinx much, at the code of losing docs in packages that don't fix their builds).

@mitya57
Copy link
Contributor

mitya57 commented Jan 12, 2025

Regarding updates to stable Python branches: good news is that we usually don’t upload patch releases of Python to Debian stable, instead we cherry-pick selected fixes. For example, Debian stable has Python 3.11.2 + patches, not 3.11.11.

Bad news is that we do it for Ubuntu: for example, Ubuntu 22.04 (jammy) was released with Python 3.10.4 but it was later updated to 3.10.12. And updating Sphinx to new major version is impossible in an Ubuntu stable release for the same reason I mentioned above (it breaks many other packages). So if this policy is accepted, Ubuntu will have to choose: not to update Python, or to update Python but revert changes that require new Sphinx, or to not build documentation at all.

@doko42 knows better about Python stable updates in Debian/Ubuntu, maybe he will correct me.

@doko42
Copy link
Member

doko42 commented Jan 13, 2025

I don't have do add more for the Debian perspective. However I do backports of Python point release in Ubuntu. That would make also backports of an additional sphinx package, and backports of additional sphinx extensions necessary. If I would have a choice, I'd like to avoid with these backports, and stay with the same major sphinx version.

@AA-Turner
Copy link
Member Author

There seems to be consensus amongst the core team to do this, and it seems to be possible for downstream redistributors (though it might mean some changes in process), so I've opened #128922 to move this forwards. Thank you to everyone for your input and feedback!

A

@AA-Turner AA-Turner moved this from Todo to In Progress in docs issues Jan 16, 2025
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jan 20, 2025
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jan 20, 2025
AA-Turner added a commit that referenced this issue Jan 20, 2025
AA-Turner added a commit that referenced this issue Jan 20, 2025
@AA-Turner
Copy link
Member Author

Thank you again to everyone for discussing this change —- I’ll leave this issue open for a little while in case something comes up, but otherwise this is now complete!

A

miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jan 20, 2025
(cherry picked from commit 4967fa6)

Co-authored-by: Adam Turner <[email protected]>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jan 20, 2025
(cherry picked from commit 4967fa6)

Co-authored-by: Adam Turner <[email protected]>
hugovk pushed a commit that referenced this issue Jan 20, 2025
GH-125722: Use long options for Sphinx (GH-129039)
(cherry picked from commit 4967fa6)

Co-authored-by: Adam Turner <[email protected]>
hugovk pushed a commit that referenced this issue Jan 20, 2025
GH-125722: Use long options for Sphinx (GH-129039)
(cherry picked from commit 4967fa6)

Co-authored-by: Adam Turner <[email protected]>
srinivasreddy pushed a commit to srinivasreddy/cpython that referenced this issue Jan 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs Documentation in the Doc dir
Projects
Status: In Progress
Development

No branches or pull requests

6 participants