[MXNET-744] Sphinx error reduction

[aaronmarkham]

Description

This PR reduces the number of Sphinx errors and warnings from 1628 to 251 on a full Sphinx build.
It reduces the warnings from a Sphinx toctree being missing from 170 to 0 on incremental Sphinx builds.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)

• Changes are complete (i.e. I finished coding on this PR)
  This PR helps cleanup some of problems to pave the way for a better understanding of how the parts work together, and how they might be updated or replaced.

• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

• Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
  Note: while this might provide a simple preview and help test the left nav and toctree features, I'd advise looking at a full site build with the versions dropdown.

Changes

1. docs/get_started folder is deleted: it only held a redirect which already being handled by .htaccess. Removing it reduces warnings/errors.
2. docs/architecture/release_note_0_9.md is deleted: it was the only legacy release note. Release notes are held on the releases page. Removing it reduces warnings/errors.
3. Sphinx's docs/conf.py updates:
   • exclude_patterns : added these to reduce warnings/errors: ['build_version_doc' (working dir), 'api/python/model.md' (depricated), 'README.md' (not needed for Sphinx's indexing)]
   • update config to the working Sphinx version 1.5.6
   • update config to 2018 copyright
   • update config to have correct project URL
4. I disabled/removed some functionality it docs/_static/js/sidebar.js. I was able to find some info about it by digging through old commits and PRs, but many of the PRs and commits have no description or commit message. I can see that the functionality may have been of some benefit before where it magically modified the output of Sphinx. However, since it wasn't documented, this caused me hours of frustration when Sphinx's documented behavior was being nullified by this client-side activity.
   • Replaying the previous sidebar layout and merging with the existing one plays havoc with the UI after one has gone through and configured the toctree for each folder. This is why I removed it. It is possible if the original authors took a look, they could update it, but for reasons I outline later, I think it would be better to remove it.
   • Adding Clojure to the nav can come in a separate PR. But since I was in the sidebar already, I added it.
5. Added new index pages where ones were missing or modified existing ones. This is generally a good practice for SEO, and drastically improves the warnings/errors situation.
   • I used the :hidden: directive for tutorials and faq as they already have custom indexes, but were the source of many warnings/errors.
   • I skipped modifying index pages that did not result in warnings/errors.
   • I used the :glob: pattern where I could to auto-index a folder. This works best when there are no subfolders to index. You can see this mostly in the tutorials section. :

.. toctree::
   :glob:
    *

Comments

• As we gear up for foreign translations of the site, I really can't see how undocumented, custom client-side overrides are going to be maintainable. We should work within the features of the site generator. I recommend we try to fix the site at it's core and as part of a regular build pipeline, and remove these kind of "magical" features.
• There is additional complexity with the versions dropdown that seems to be covered in the sidebar logic. Disabling this client-side hack might result in uncovering more issues and entanglements. I'd appreciate any SMEs to assist figuring out solutions that can help simply things and get us ready for internationalization of the site.

Usage

These changes gets triggered in a regular make docs, but you can run just the Sphinx section of the docs pipeline by commenting out these in docs/mxdoc.py:

#app.connect("builder-inited", generate_doxygen)
#app.connect("builder-inited", build_scala_docs)
#app.connect("builder-inited", build_clojure_docs)

Then run (from /docs):

make html

[aaronmarkham]

@ThomasDelteil @thomelane @sandeep-krishnamurthy @kevinthesun @piiswrong @mli @nswamy @marcoabreu - you all use this, or have used it... please let me know if you have any suggestions.

[szha]

AWESOME!

[ThomasDelteil]

Thanks for cleaning this up @aaronmarkham

[aaronmarkham]

I found an issue with the versions and the sidebar.js. The sidebar is missing in the API pages with older versions (non-master). Let's hold off merging until I can figure out what magic it was doing.

[aaronmarkham]

UPDATE:
Part of the problem is changes I make here are not viewable in the full site build pipeline as that checks out from specific branches on the main repo and not my fork. Then each legacy version only uses the configurations it has and not any updates I make here.
So... while I can fix the current build, the builds for every other version are still buggy as hell. 😖

I'm working on the tooling now so that you can supply logic from your fork/branch and supply that as an arbitrary version in the build. This process also is belabored by every version and every API build which makes for test cycles of about 40 minutes. I'm updating the build tools to turn on/off documentation sets via a config file instead of how they're hard-coded in the Sphinx plugin file, mxdoc.py. I've got it down to 18 minutes for all versions.

This has led to certain realizations.
💡 We're probably going to have to refactor the docs build process, so that we have better control.
💡 Each version should be built separately and maintained separately, rather than an overlapping pipeline.
💡 Each version should be built with configurations held in master, or...
💡 We're probably going to have to make some commits to the old version branches to fix legacy bugs, so that we're not doing hot patches during every build.
💡 Maybe we can cache old versions and not have to rebuild every time. Then builds can be done in a few minutes.
💡 And finally, maybe I can actually debug whatever is going on with this Sphinx theme! Or if anyone really knows Sphinx theme customization and how this code works, they can help address why the left navigation behaves unexpectedly now that index files and table of contents are being supplied as per Sphinx's documentation. Basically, it seems to copy state from previous views and when you toggle between versions --> shenanigans.

Any help would be appreciated. 🙏

[aaronmarkham]

I rebased this on top of #11990, so I get the benefit of the docs build tools refactor.
Also, the Jenkinsfile in this PR will need to be reset to production settings once everything is approved.

[aaronmarkham]

Using the new build pipeline, I created a preview website with the versions dropdown to show what happens with the toctree updates. Note that the dropdown show my fork's branch, which was used to create one of the versions of the website, and as the default version. Here's a screenshot showing the left side nav is unexpectedly different. Not bad really. Just not what we had before. It now lists the APIs which is coming from the toctree.

Here's the view if you switch to the 1.2.1 version:

I'm not sure how to make the updated version where all of the toctrees are in place, to look like the old version. Or, maybe the old version was always buggy, but we didn't realize it since it worked despite all of the Sphinx errors.

If no one is opposed to this new behavior for the left navigation, then I would say this PR is ready.

[lebeg]

Looks good

[aaronmarkham]

Need to rebase and test after recent updates! Let me do that before merging. Thanks!

[aaronmarkham]

This is viewable here: http://34.201.8.176/
Looks good to me.
I should have noted earlier that this only fixes the errors in master and versions going forward. There are still a ton of errors when we build the old versions of the docs and website. I think this patch could be applied to each old branch with good success if that's of interest.