Some way to put documentation on separate pages like sphinx.ext.autosummary

[asmeurer]

Is there a way to put the documentation for each function/method on a separate page like you can do with sphinx.ext.autosummary? I have been trying to figure out if it's possible to do it by somehow mixing autodocsumm and autosummary, but I'm not seeing a straightforward way to do it.

My main motivation here is that I like that autodocsumm uses autodoc to determine which members are included. autosummary uses its own method, which is very coarse compared to autodoc (e.g., autosummary doesn't seem to have a way to exclude undocumented or private methods, even by editing the template).

[Chilipp]

Hey @asmeurer ! Thanks for your input! That does indeed sound like a useful feature 😃 autosummary does generate these files in a pre-processing step, so outside of the build process, which is something we cannot do with autodocsumm. So my naive idea would be to implement something that is called with the builder-inited signal. Then you go through all files and look for automodule and autoclass directives and generate the ones you need (maybe you can even use autosummary there).

Would you have time and resources to start a PR on this? I'd be happy to contribute

[asmeurer]

Yes, I think I can give this a shot, although I'm a little afraid it won't be easy. I'll have to see how well I can make sense of the code. I've written a Sphinx extension before, but I know that it can be complicated. My guess is that we could probably crib some code from autosummary, but it won't be as simple as just calling out to it. The reason is that autosummary simply lacks the parameters in its templating to do this properly (which is the whole issue in the first place, otherwise I could just make a custom template to do what I want).

By the way, I realized another limitation of autosummary (or apparent limitation, I really have no idea if there's a way around this). You can't really control what it uses for the title, and more importantly, the URL of a page. It just uses the fully qualified name of whatever function. You can list the name of the functions like ~function or even using currentmodule, but it still uses the fully qualified name in the function. You can change this in the template, but not on a per-function basis. This basically forces the fully qualified name of every function to become part of the public API, whether you want it to or not, because it's forced to be there on the page, and because changing a submodule name or moving a function around would break your URLs.

For SymPy, some modules are exposed at the top level, so shouldn't use a fully qualified name, and some aren't, so should.

[Chilipp]

Alright! Thanks for the contribution then already 😃 Please let me know if and how I can help you and when you start working on it. It shouldn't be too hard to overcome this file-naming issue.

[asmeurer]

I did some looking at the code, but haven't written anything yet. I also wanted to see if it would be possible to avoid writing files to disk at all, but as far as I can tell, this is impossible, or at least it would be extremely difficult. Sphinx seems to have a pretty hard assumption that every document corresponds to a file on disk. (the only reason I wanted to look at this is that it would be nice if I didn't have to add a separate URL fragment for the autosummary pages, corresponding to the toctree directory, but this doesn't seem like an easy enough thing to be worth pursuing).

Also it seems like I was correct with my gut feeling about being able to reuse some autosummary code but not being able to just call out to it directly. The autosummary code buries its logic in functions defined within functions, and seems to hardcode a lot of things which we would want to do differently.

[asmeurer]

The one piece of the puzzle I'm still trying to figure out is how to hook into the autodoc machinery to get a list of all the pages that should be generated at the builder-inited stage. It seems like I would need to manually parse each autodoc directive and extract its contents, then somehow inject that into the appropriate autodoc functions. Do you have any thoughts on an easy way to do this?

It seems like the Sphinx build process isn't really designed for this sort of thing. It would be nice if we could just inject documents after the parsing stage, or if we could otherwise run the build in two stages, one to parse the autodoc and extract the pages to ge generated, and another to rebuild everything with those pages generated. But if it's possible to do either of these things, it isn't clear to me how.

[Chilipp]

Hmm, no idea. Isn't autosummary doing the same with it's generate command? See here. But yes, this seems quite complicated indeed...

[asmeurer]

So I've been looking into how to parse the autodoc directive. As far as I can tell, you more or less have to use regular expressions like autosummary is doing. It's not really possible to use the Sphinx parser at the builder-inited state, because it hasn't set up all the internal state that it needs to call the parser yet. This is probably fine. I think it might just mean that it won't work when mixed with some other extensions or that we might miss some corner cases.

The thing I haven't figured out yet, though, is how to actually call into the autodoc code to get the list of member names. It seems like I can't do much with the Documenter (or AutosummDocumenter) class without instantiating it, which requires creating a DocumenterBridge object. This object seems to involve things that don't exist yet at the builder-inited stage (though I could be wrong about that). Any thoughts on how I might get that working?

I think I understand how to get the rest of the pieces working, by more or less copying what autosummary is doing. I don't have any code yet, as I want to make sure I know how to get this last bit working first (getting the list of names from autodoc). If I can't get that to work, then this whole approach won't work and we'll either have to think of something else.

[asmeurer]

@chrisjsewell @choldgraf, I know this isn't exactly related to what you are working on, but I know you are both pretty knowledgeable about the Sphinx internals. I wonder if you have any thoughts or suggestions on how I might implement something like this.