Docs build error with Sphinx 3.0 due to invalid C declaration

[tirkarthi]

BPO	40204
Nosy	@birkenfeld, @rhettinger, @terryjreedy, @doko42, @vstinner, @benjaminp, @ned-deily, @ezio-melotti, @merwok, @skrah, @serhiy-storchaka, @willingc, @JulienPalard, @miss-islington, @tirkarthi, @m-aciek
PRs	bpo-40204: Update documentation formatting for Sphinx 3.0 #19397 [3.8] bpo-40204: Pin Sphinx version to 1.8.2 in Doc/Makefile #19442 [3.7] bpo-40204: Pin Sphinx version to 1.8.2 in Doc/Makefile (GH-19442) #19443 bpo-40204: Pin Sphinx version to 2.3.1 in Doc/Makefile. #21141 [3.9] bpo-40204: Pin Sphinx version to 2.3.1 in Doc/Makefile. (GH-21141) #21146 [3.8] bpo-40204: Pin Sphinx version to 2.3.1 in Doc/Makefile. (GH-21141) #21147 bpo-40204: Allow pre-Sphinx 3 syntax in the doc #21844 bpo-40204, doc: Fix syntax of C variables #21846 bpo-40204: Fix duplicates in the documentation #21857 bpo-40204: Fix Sphinx sytanx in howto/instrumentation.rst #21858 bpo-40204: Add :noindex: in the documentation #21859 bpo-40204: Fix reference to terms in the doc #21865 bpo-40204: Fix duplicated productionlist names in the doc #21900 [3.9] bpo-40204: Allow pre-Sphinx 3 syntax in the doc (GH-21844) #21901 [3.8] bpo-40204: Allow pre-Sphinx 3 syntax in the doc (GH-21844) (GH-21901) #21924 [3.8] bpo-40204: Allow pre-Sphinx 3 syntax in the doc (GH-21844) (GH-21901) #21928 bpo-40204: Update Sphinx to version 3.2.1 in Doc/Makefile #22043

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-08-20.11:33:09.064>
created_at = <Date 2020-04-06.14:32:48.165>
labels = ['type-bug', '3.8', '3.9', '3.10', 'docs']
title = 'Docs build error with Sphinx 3.0 due to invalid C declaration'
updated_at = <Date 2020-09-02.10:29:35.521>
user = 'https://github.com/tirkarthi'

bugs.python.org fields:

activity = <Date 2020-09-02.10:29:35.521>
actor = 'vstinner'
assignee = 'docs@python'
closed = True
closed_date = <Date 2020-08-20.11:33:09.064>
closer = 'vstinner'
components = ['Documentation']
creation = <Date 2020-04-06.14:32:48.165>
creator = 'xtreak'
dependencies = []
files = []
hgrepos = []
issue_num = 40204
keywords = ['patch']
message_count = 42.0
messages = ['365858', '365872', '365876', '365895', '366021', '366022', '366023', '366082', '366090', '366091', '366096', '366097', '366172', '371790', '372262', '372265', '372271', '372292', '372347', '372348', '372353', '372698', '372701', '372702', '372704', '372858', '373083', '374736', '375254', '375281', '375320', '375321', '375335', '375341', '375343', '375392', '375527', '375529', '375661', '375703', '375704', '376223']
nosy_count = 18.0
nosy_names = ['georg.brandl', 'rhettinger', 'terry.reedy', 'doko', 'vstinner', 'benjamin.peterson', 'ned.deily', 'ezio.melotti', 'eric.araujo', 'skrah', 'docs@python', 'python-dev', 'serhiy.storchaka', 'willingc', 'mdk', 'miss-islington', 'xtreak', 'Maciej Olko']
pr_nums = ['19397', '19442', '19443', '21141', '21146', '21147', '21844', '21846', '21857', '21858', '21859', '21865', '21900', '21901', '21924', '21928', '22043']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40204'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[tirkarthi]

The following error is caused in Docs build for a 3.8 backport since sphinx is ran with warnings. Sphinx released 3.0 on April 6. The last successful build on master uses Sphinx 2.2.0 [0]. My guess is sphinx new version possibly breaking the build on Python 3.8 where it's not pinned to use 2.2.0 pulling the latest version. The changelog for Sphinx has below note :

https://www.sphinx-doc.org/en/master/changes.html#release-3-0-0-released-apr-06-2020

The C domain has been rewritten, with additional directives and roles. The existing ones are now more strict, resulting in new warnings.

Python 3.8 and Python 3.7 doesn't have Sphinx pinned to 2.2.0 while master does.

Python 3.8 Docs makefile :

cpython/Doc/Makefile

Line 146 in f7b0259

$(VENVDIR)/bin/python3 -m pip install -U Sphinx blurb python-docs-theme

Failed build :

https://github.com/python/cpython/pull/19388/checks?check_run_id=563053793#step:7:46

Error :

Warning, treated as error:
/home/runner/work/cpython/cpython/Doc/c-api/buffer.rst:92:Error in declarator or parameters
Invalid C declaration: Expected identifier in nested name. [error at 5]
void \*buf
-----^
Makefile:49: recipe for target 'build' failed
make[1]: *** [build] Error 2

[0] https://github.com/python/cpython/runs/564123625#step:6:24

[vstinner]

It sounds dangerous to not pin the Sphinx version in our CI :-/ Another issue caused by CI configuration stored at the same place than the code:
https://mail.python.org/archives/list/[email protected]/thread/WEU5CQKIA4LIHWHT53YA7HHNUY5H2FUT/

[serhiy-storchaka]

Maybe copy the code for deprecated and removed features to Doc/tools/extensions?

[tirkarthi]

The sphinx version is not pinned in 3.7, 3.6, 3.5 and 2.7 branches too for Doc/Makefile that can cause error on someone trying it out locally. They are pinned in .travis.yml and .azure-pipelines configurations.

[vstinner]

Python 3.8 and Python 3.7 doesn't have Sphinx pinned to 2.2.0 while master does.

In 3.8, .azure-pipelines/docs-steps.yml contains:

• script: python -m pip install sphinx==1.8.2 blurb python-docs-theme

and .travis.yml contains:

• python -m pip install sphinx==1.8.2 blurb python-docs-theme

For example, the Sphinx version was changed from 1.8.1 to 1.8.2 in the CI configuration by:

commit 7f4ba4a
Author: Julien Palard <[email protected]>
Date: Sat Nov 24 11:35:21 2018 +0100

Doc: Bump sphinx. (GH-10676)

--

I guess that you're talking about Doc/Makefile which uses "Sphinx" in 3.8 but "Sphinx==2.2.0" in master. Code in 3.8:

venv:
$(PYTHON) -m venv $(VENVDIR)
$(VENVDIR)/bin/python3 -m pip install -U pip setuptools
$(VENVDIR)/bin/python3 -m pip install -U Sphinx blurb python-docs-theme
@echo "The venv has been created in the $(VENVDIR) directory"

[vstinner]

I guess that you're talking about Doc/Makefile which uses "Sphinx" in 3.8 but "Sphinx==2.2.0" in master.

I wrote PR 19442 to pin Sphinx version to 1.8.2 in Doc/Makefile.

[vstinner]

New changeset 37a257c by Victor Stinner in branch '3.8':
bpo-40204: Pin Sphinx version to 1.8.2 in Doc/Makefile (GH-19442)
37a257c

[tirkarthi]

I have filed an issue upstream and it's fixed. The release of 3.0.1 is planned in few days and could help for other branches. But it would be nice to see the version pinned to avoid these problems in future. Upstream report : sphinx-doc/sphinx#7423 (comment)

[vstinner]

New changeset 9e5f159 by Victor Stinner in branch '3.7':
bpo-40204: Pin Sphinx version to 1.8.2 in Doc/Makefile (GH-19442) (GH-19443)
9e5f159

[ned-deily]

Why are we pinning to 1.8.2 when the official docs builds for all releases are currently using 2.3.1? (see, for example, https://docs.python.org/3.8/ at bottom right corner)

[vstinner]

Why are we pinning to 1.8.2 when the official docs builds for all releases are currently using 2.3.1? (see, for example, https://docs.python.org/3.8/ at bottom right corner)

First of all, to repair the CI :-) Before my change, it was no longer possible to merge any change in 3.8! (I didn't check for 3.7, but I expect that it wasn't possible neither).

Second, 1.8.2 version comes from the CI configuration:

$ grep -i sphinx== .travis.yml .azure-pipelines/* Doc/Makefile 
.travis.yml:        - python -m pip install sphinx==1.8.2 blurb
.azure-pipelines/docs-steps.yml:- script: python -m pip install sphinx==1.8.2 blurb python-docs-theme
Doc/Makefile:	$(VENVDIR)/bin/python3 -m pip install -U Sphinx==1.8.2 blurb

If Sphinx version is changed in Doc/Makefile, I would prefer to also update the version in the CI configuration.

I also vaguely recall discussions about issues with newer Sphinx. I don't recall if it was 2.0 or another version. Julien Palard may recall that better than me! Was it bpo-35472?

--

I don't really care of the Sphinx version, I only care about working CI and consistency between all pinned versions ;-)

--

By the way, maybe Doc/Makefile could use the latest Sphinx version by default, and the Python CI could use a pinned version. The problem is that the Sphinx version cannot be configured currently when running "make venv".

Oh, .travis.yml is differecen between 3.7 and 3.8.

3.8 and master use "make -C Doc/ PYTHON=../python venv".

3.7 doesn't use "make venv" but:

before_script:
- cd Doc
# Sphinx is pinned so that new versions that introduce new warnings won't suddenly cause build failures.
# (Updating the version is fine as long as no warnings are raised by doing so.)
- python -m pip install sphinx==1.8.2 blurb
script:
- make check suspicious html SPHINXOPTS="-q -W -j4"

Maybe 3.7 should mimick what is done in 3.8 and master to ease maintenance. I don't know.

[ned-deily]

I agree that it should be easier to keep them all in sync. But my point is that the on-going official doc builds (some multi[ple times per day) for all of the active versions (2.7 and 3.6 through 3.9) are using 2.3.1 so we should be doing CI against that version as well. The docs builds are configured in https://github.com/python/docsbuild-scripts. So whenever requirements.in is update there, the CI versions should be updated as well. Even better would be someway to auto sync them.

Julien, any ideas on this?

[terryjreedy]

A few days ago, 3.8 backports failed while 3.7 backports merged. Thank you for the fix, even if temporary.

I just noticed that the doc main page ends with "Created using Sphinx 2.3.1." I plan to switch my local sphinx to that version.

[ned-deily]

New changeset 0d70a98 by Ned Deily in branch '3.7':
bpo-40204: Pin Sphinx version to 2.3.1 in Doc/Makefile
0d70a98

[vstinner]

I look into this issue. It's quite complicated :-(

The Sphinx 3.0.0b1 changelog says:

• The C domain has been rewritten, with additional directives and roles. The existing ones are now more strict, resulting in new warnings.

• The C domain has been rewritten adding for example:

  • Cross-referencing respecting the current scope.
  • Possible to document anonymous entities.
  • More specific directives and roles for each type of entitiy, e.g., handling scoping of enumerators.
  • New role c:expr for rendering expressions and types in text.

The Python documentation has many issues:

• Doc/howto/instrumentation.rst: ".. c:function:: function__entry(str filename, str funcname, int lineno)" is not valid
  => replace "str" with "char*" and explain that "char*" must be read as "str"... The documentation is about DTrace functions, it's not C code.

• Doc/library/configparser.rst: "unique" is declared twice
  => the second declaration must have :noindex: marker

• ".. c:function:: PyObject* PyObject_CallMethodObjArgs(PyObject *obj, PyObject *name, ..., NULL)": the "..., NULL" part is invalid in C => remove ", NULL"

• .. c:var:: PY_VECTORCALL_ARGUMENTS_OFFSET: a variable must be declared with its type, but PY_VECTORCALL_ARGUMENTS_OFFSET is macro
  => use ".. c:macro::" instead

• ".. c:var:: Py_BytesWarningFlag": the type is missing
  => ".. c:var:: int Py_BytesWarningFlag"

• Doc/c-api/call.rst: vectorcallfunc documentation is not properly indented.
  => fix indentation

• Doc/c-api/buffer.rst: the table with ".. c:macro:: PyBUF_INDIRECT" is wrong. Sphinx fails because PyBUF_SIMPLE is declared twice.
  => use ":c:macro:`PyBUF_INDIRECT`" instead

• :c:type:`PyObject\` is invalid
  => use :c:type:`PyObject`

• ... there are many other errors :-(

PR 19397 fix some errors, but not all of them.

[vstinner]

Sphinx documentation of the C domain:
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#the-c-domain

[birkenfeld]

Don't hesitate to send requests back to the Sphinx tracker if any changes appear to be too restrictive to you. It is a documentation tool, not a compiler.

[skrah]

"""

• Doc/c-api/buffer.rst: the table with ".. c:macro:: PyBUF_INDIRECT" is wrong. Sphinx fails because PyBUF_SIMPLE is declared twice.
  => use ":c:macro:`PyBUF_INDIRECT`" instead
  """

Hmm, grep shows:

c-api/buffer.rst:123: most cases such a request will be :c:macro:`PyBUF_SIMPLE` or :c:macro:`PyBUF_WRITABLE`.
c-api/buffer.rst:144: If :c:member:`~Py_buffer.shape` is ``NULL`` as a result of a :c:macro:`PyBUF_SIMPLE`
c-api/buffer.rst:255:Since :c:macro:`PyBUF_SIMPLE` is defined as 0, :c:macro:`PyBUF_WRITABLE`
c-api/buffer.rst:258::c:macro:`PyBUF_FORMAT` can be \|'d to any of the flags except :c:macro:`PyBUF_SIMPLE`.
c-api/buffer.rst:280:| .. c:macro:: PyBUF_SIMPLE | NULL | NULL | NULL |

So I see a single definition ans several references. Whether the definition may occur in a table is more a question for Georg. :)

It worked up to now and is quite useful, because you can jump directly into the correct row.

Escaping the "\*" was necessary at the time for vim syntax highlighting. I think it is no longer needed. So everything has a reason :)

[ned-deily]

New changeset 589e8fe by Ned Deily in branch 'master':
bpo-40204: Pin Sphinx version to 2.3.1 in [Doc/Makefile](https://github.com/python/cpython/blob/main/Doc/Makefile). (GH-21141)
589e8fe

[miss-islington]

New changeset 16e79a4 by Miss Islington (bot) in branch '3.9':
bpo-40204: Pin Sphinx version to 2.3.1 in [Doc/Makefile](https://github.com/python/cpython/blob/main/Doc/Makefile). (GH-21141)
16e79a4

[ned-deily]

New changeset 7318f0a by Ned Deily in branch '3.8':
bpo-40204: Pin Sphinx version to 2.3.1 in [Doc/Makefile](https://github.com/python/cpython/blob/main/Doc/Makefile). (GH-21141) (GH-21147)
7318f0a

[vstinner]

This issue prevents to upgrade Sphinx to Sphinx 3 in Fedora Rawhide, at least it breaks the python3-docs package which is Python 3.9 documentation:
https://bugzilla.redhat.com/show_bug.cgi?id=1823898

[vstinner]

Making the Python documentation compatible with Sphinx 3.0 is causing multiple blocker issues.

(*) The first major problem is that new C domain markup like :c:struct: and :expr: are not recognized by Sphinx 2.4.4, but Sphinx 3.0 requires them. For example, a ".. c:struct:" section is completely hidden: treated as a comment.

I don't think that we can afford to drop Sphinx < 3 support since no Linux distribution uses Sphinx 3 yet. I see two options:

• Stay at Sphinx forever
• Backport the ":c:struct:" and ":c:expr:" markups somehow in Sphinx 2 using our CPython Sphinx extensions
• Update Sphinx 3 to tolerate our "incorrect" documentation using :c:type: and :c:type:`PyObject\*` (invalid C syntax), maybe as an opt-in option
• Update Sphinx 2 to support Sphinx 3 new syntax: something like Python 2.7 which backported a few Python 3 features like b"bytes string" syntax

(*) The second major question is: should we backport these changes to Python 3.8 and 3.9 branches? I consider that "supporting Sphinx 3" is a fix in the Python build system, and usually we do backport such changes, even if they can be seen as "new features". For me, it falls into the same category than supporting a newer version of C compiler.

PR 19397 makes tons of small changes. If we don't backport it, it will be very likely very painful to backport documentation changes. Julien Palard (who works on the documentation) suggests to backport the change. Usually, documentation changes are applied to all branches, not only master.

Again, right now, if we backport these changes, we would drop Sphinx 2 support.

[vstinner]

In short, the current status is that IMHO it is not acceptable to support Sphinx 3. We cannot afford to break support with Sphinx 2.

[vstinner]

I reported the issue to Sphinx:
"C domain changes of Sphinx 3 prevent to write doc compatible with Sphinx 2 and Sphinx 3"
sphinx-doc/sphinx#7899

[vstinner]

Using sphinx-doc/sphinx#7905 but without -W, I'm able to build the unmodified Python documentation with Sphinx 3!

I tried c_allow_pre_v3=1.

The PR was updated to add a second c_warn_on_allowed_pre_v3=0 option so we can keep -W.

[doko42]

please note that pinning usually is not a solution for Linux distributions. Yes, the most wanted fix would be to fix sphinx 3.x not to break compatibility with 2.x. Or limit 3.9 to 2.x features for now.

[vstinner]

sphinx-doc/sphinx#7905 has been merged and will be part of the next Sphinx 3.2 release.

[doko42]

3.9.0 rc1 fails to build the docs with Sphinx 3.2.0, even with setting

c_allow_pre_v3 = True
c_warn_on_allowed_pre_v3 = False

Warning, treated as error:
/packages/python/3.9/python3.9-3.9.0~rc1/Doc/c-api/buffer.rst:92:Error in declarator or parameters
Invalid C declaration: Expected identifier in nested name. [error at 5]
void \*buf
-----^
make[1]: *** [Makefile:52: build] Error 2

With 3.2.0, you still need to build without -W

[vstinner]

New changeset 423e77d by Victor Stinner in branch 'master':
bpo-40204: Allow pre-Sphinx 3 syntax in the doc (GH-21844)
423e77d

[vstinner]

New changeset 43577c0 by Victor Stinner in branch 'master':
bpo-40204: Fix Sphinx sytanx in howto/instrumentation.rst (GH-21858)
43577c0

[vstinner]

New changeset 46d10b1 by Victor Stinner in branch 'master':
bpo-40204: Fix duplicates in the documentation (GH-21857)
46d10b1

[vstinner]

New changeset d3ded08 by Victor Stinner in branch 'master':
bpo-40204: Add :noindex: in the documentation (GH-21859)
d3ded08

[vstinner]

New changeset 474652f by Victor Stinner in branch 'master':
bpo-40204, doc: Fix syntax of C variables (GH-21846)
474652f

[vstinner]

With PR 21865, there are only two remaining warnings:

Doc/library/string.rst:311: WARNING: duplicate token description of sf:format_spec, other instance in library/string
Doc/reference/introduction.rst:96: WARNING: duplicate token description of *:name, other instance in reference/expressions

See bpo-35293 for RemovedInSphinx40Warning (which are already prevent with Sphinx 2).

[vstinner]

New changeset bb0b085 by Victor Stinner in branch 'master':
bpo-40204: Fix reference to terms in the doc (GH-21865)
bb0b085

[vstinner]

New changeset 1abeda8 by Victor Stinner in branch 'master':
bpo-40204: Fix duplicated productionlist names in the doc (GH-21900)
1abeda8

[vstinner]

I'm now able to build the Python documentation with Sphinx 3.2.1 without modifying the Doc/Makefile, so using -W option (treat warnings as errors): there are no more Sphinx 3 warnings.

When Sphinx 3 will be more widely available (ex: in Linux distributions), we will be able to consider removing c_allow_pre_v3=True and c_warn_on_allowed_pre_v3=False in doc/conf.py. For that, we should update the documentation to use the Sphinx 3 syntax, see PR 19397 written by Jakob Lykke Andersen.

For now, I prefer to keep Sphinx 2 support and so keep Sphinx 2 syntax in the C domain, since it allows supporting Sphinx 2 and Sphinx 3.

[vstinner]

New changeset 8f88190 by Victor Stinner in branch '3.9':
[3.9] bpo-40204: Allow pre-Sphinx 3 syntax in the doc (GH-21844) (GH-21901)
8f88190

[vstinner]

New changeset 7d0fef5 by Victor Stinner in branch '3.8':
bpo-40204: Allow pre-Sphinx 3 syntax in the doc (GH-21844) (GH-21901) (GH-21928)
7d0fef5

[vstinner]

I merged my changes to add Sphinx 3.2 and newer support into 3.8, 3.9 and master branches.

I close the issue.

Sadly, it seems like it's not possible to keep Sphinx 2 support if we add Sphinx 3.0 and Sphinx 3.1 support. For now, only Sphinx 2 and Sphinx 3.2+ are supported: 3.0 and 3.1 are not supported.

[vstinner]

New changeset c0d5c13 by Victor Stinner in branch 'master':
bpo-40204: Update Sphinx to version 3.2.1 in Doc/Makefile (GH-22043)
c0d5c13