autodoc: Use type stub files when documenting native modules

[AA-Turner]

Purpose

This builds on work originally proposed by @alliefitter in #4824. Autodoc now attempts to load a .pyi stub file for any native modules being documented.

Currently this will become the default behaviour, but we could put it behind an environment variable if it might be disruptive (SPHINX_AUTODOC_USE_NATIVE_STUB_FILES?).

cc commenters from original threads: @JosXa, @hoefling, @Vanuan, @cmpute, @brenthuisman, @kdschlosser, @jonnor.

References

• Closes Added support for reading .pyi files. #4824
• Closes Use PEP 484 stub files when documenting native extensions with autodoc #7630
• https://github.com/bayashi-cl/stubdoc (Use PEP 484 stub files when documenting native extensions with autodoc #7630 (comment))

A

[electric-coder]

This is a ground shaking improvement for the whole ecosystem. I have to ask:

• Will Sphinx only extract types or also docstrings from the .pyi (because apparently you can also outsource your docstrings to the stub files and that would make for much cleaner .py)?

Edit: @hoefling points out problems with putting docstrings in stub files

[kdschlosser]

OH YES!!! This would be awesome to have available... FINALLY!!!

Fingers crossed it gets merged.

[kdschlosser]

as a note. if there is a stub file then the docstrings should be in that file. The purpose for the stub file is not just for typing. It is also to remove code that doesn't need to be apart of the running code.

The only place where I can see this being an issue is with python's built in help() function. I have never tested to see if help() pulls from a stub file if it is available. If it doesn't anmd it only pulls from the source file then all that needs to bed one is moving the types and docstrings into the source file and then help() would function like it should.

that is the only thing I can see as being a problem other than dynamic creation of classes and in that case there would not be a stub file unless one is generated.

[electric-coder]

The only place where I can see this being an issue is with (...)

Last I checked mypy stops type checking your .py internal annotations if you have a stub file for it, see python/mypy#5028 so that would now be the main blocking factor. If you notice the mypy issue is 7 years old and still with no solution in sight.

Likely we'll get several more years of new typing PEPs without proper support for them.