Handle dots in id attributes for Python module members with custom HTMLTranslator

[bheberlein]

I'll preface this by saying I am not entirely sure if this is more properly a request for PST or for Sphinx.

When I view my generated package index from sphinx-apidoc, I'd like to have a responsive secondary sidebar that expands & collapses individual submodules in the navigation TOC. However, because of Python namespace conventions, the id attributes of generated HTML <section> & <dt> tags contain dots, e.g. <section id="mypackage.submodule1"> ... </section> and <dt id="mypackage.submodule1.func_a"> ... </dt>. Because of this, ScrollSpy does not detect these tags as targets, and the auto-expand & auto-collapse functionality we see on the sidebars of 'normal' pages do not work here.

Because of PST's reliance on ScrollSpy for responsive layouts, and because un-escaped . in a CSS id attribute is incompatible with ScrollSpy (and not really valid CSS anyway) it seems to make sense to handle this for documented package members.

I thought sphinx-build might provide options to handle this out-of-the-box, but I wasn't able to discover any way to configure this.

Sketch of a solution

The id attributes can be made valid by either escaping with a backslash (mypackage\.submodule1) or by substituting some other character, e.g. an underscore (mypackage_submodule1). I opted for the latter here, but it's really a question of preference. Maybe escaping the . would be more faithful to the Python namespace conventions (though I consider the underscore a bit nicer myself).

I'll note that it may also be possible to achieve the desired results by modifying bootstrap.js, though JavaScript is not my area of expertise. It would be necessary to modify the getSelectorFromElement() function, which returns null because querySelector() fails on IDs containing dots.

Anyway, I believe we can also handle this with a custom HTMLTranslator class that implements the following methods:

from sphinx.writers.html import HTMLTranslator


class PyModule HTMLTranslator(HTMLTranslator):

    """Custom HTML translator to convert dotted module names into underscore-separated ``id`` & ``href`` attributes."""
    
    def visit_section(self, node):
        """Override default behavior to replace dots with underscores in the ``id`` attribute of documented modules."""
        # NOTE: This will modify the `id` attribute of HTML `<section>` tags, e.g. where Python modules are documented
        if 'ids' in node:
            node['ids'] = [id_.replace('.', '_') for id_ in node['ids']]
        super().visit_section(node)

    def visit_reference(self, node):
        """Override default behavior to replace dots with underscores in ``href`` attributes."""
        # NOTE: This will modify the `href` attribute of HTML `<a>` tags, e.g. the sidebar links
        if 'refuri' in node:
            if '.' in node['refuri']:
                node['refuri'] = node['refuri'].replace('.', '_')
        super().visit_reference(node)

    def visit_desc_signature(self, node):
        """Replace dots with underscores in the ``id`` attribute of documented Python function signatures."""
        # NOTE: This will modify the `id` attribute of HTML `<dt>` tags, where Python functions are documented
        ids = node['ids']
        for i, id in enumerate(ids):
            ids[i] = id.replace('.', '_')
        super().visit_desc_signature(node)

Of course, PST already implements BootstrapHTML5TranslatorMixin to do some customization of the HTML translation process. So the above methods can be added directly to this mixin class to achieve my desired functionality. I am not sure if adding these would mess things up anywhere else, but I haven't found any such downsides.

I'm curious what others think about this. Is it better for the end-user to implement this in their own custom HTMLTranslator class? If so, how would this even work? I ran into some difficulty with it since it seems to conflict with PST's own initialization routine (see below).

Attempt to implement solution on the user end

First I tried to add the following to my conf.py, along with the custom class definition shown above:

html_translator_class = PyModuleHTMLTranslator

This does not work, maybe because it is overridden by PST's custom translator? The documentation builds successfully, but I don't see any changes reflected in the output HTML. So I tried

def setup(app):
    app.set_translator('html', PyModuleHTMLTranslator, override=True)

but this gives an error:

Extension error:
Translator for 'html' already exists

Finally I tried

def setup(app):
    app.setup_extension('pydata_sphinx_theme')
    app.set_translator('html', PyModuleHTMLTranslator, override=True)

but then I get

Theme error:
An error happened in rendering the page changelog.
Reason: TemplateNotFound('../components/search-field.html')
make: *** [html] Error 2

I've never tried to implement a custom translator for Sphinx, and I'll admit I don't really know where to go from here. I'll note that this might suggest an issue with PST — how are end-users supposed to implement a custom translator class if they want to? Is there a way to extend the PST custom BootstrapHTML5Translator class? If so, can we add some explanation of how to do this to the docs? I can open a separate issue for this if appropriate.

It might also make sense to open an issue with the Sphinx maintainers. But in the interest of supporting a responsive theme that is compatible with existing or older Sphinx versions, I thought I would bring this up here first.

Proposed changes to PST

Add the custom visit_section, visit_reference and visit_desc_signature methods shown above to BootstrapHTML5TranslatorMixin. I tried it, and everything works just as desired.

[12rambau]

this issue is bugging me for so long (#1026) thank you so much for bringing the solution on a plate !

I'm +10000 to make it part of our translator mixin. Would you have time to start a PR ?

[bheberlein]

Yes, no problem! I can submit a PR this week.

[12rambau]

I think the merge of your proposed PR solve the issue? Could you confirm and if not the case let us know what remain to be done?