Config: support templates to allow the reuse of config across multiple projects

[eirikurj]

What's the problem this feature will solve?

Our organization has a number of repositories, each building its own documentation on RTD. With the recent config file requirement, we are faced with adding a .readthedocs.yaml file to all our repositories, which in our case, is going to be mostly identical between repositories. However, this results in an unnecessary amount of duplication, not to mention future maintenance. For example, changing/updating a single build parameter, all repository config files need to be updated. Ideally, there should be a centrally managed .readthedocs.yaml with common parameters, which then each repository can extend and configure based on its need.

Describe the solution you'd like

I am sure there are other approaches, but adopting a similar templating solution as Azure provides would be ideal. This approach we are already using when building and testing our codebase with great success.

Any support or suggestions on how to solve or approach this are greatly appreciated.

[humitos]

Hi @eirikurj. Thanks for opening this issue.

We had a similar issue for multiple services and tools we use in the exact same way for many repositories. We arrived to a pattern that involves using a Git submodule with all the common files (https://github.com/readthedocs/common/) that are shared among all our repositories. That's a solution you can use for Read the Docs as well.

That covers the case where all the YAML files are identical, tho. It doesn't cover the ability to replace some chunks/values from the YAML file. For that case, would be enough to support "Environment Variables" in the config file?

Example

version: 2

build:
  os: ${READTHEDOCS_OS}
  tools:
    python: ${READTHEDOCS_PYTHON_VERSION}

[eirikurj]

Hi @humitos and thanks for your suggestions! This approach does indeed reduce duplicated config file, but we are not sure if this fits our needs and will reduce maintenance, which is the primary objective. Using a git submodule feels a bit excessive for a single config file. The fact that its tied to a specific commit is not great either, since in our case we always want the latest. Granted, this could possibly be mitigated by tracking a branch and after init, updating with, e.g., git submodule update --remote, which should bring the submodule to latest commit. However, this feel unnecessarily convoluted.

The use of environmental variables in the config file is interesting and does offer some flexibility. However, we do see some shortcomings, especially in terms of maintenance effort. Since there are no default values defined in the shared file, you would need to define and configure every variable for all projects using the RTD dashboard. This adds lots of maintenance when updating, and the effort required is the same as just updating local config files. Furthermore, this creates a mix of config file and dashboard configuration for each project, but ideally we would like to minimize the dashboard usage, and store the config only as code.

Overall, using a template solution, where the config can be extended with parameters, similar to the previously mentioned Azure, checks all boxes, but we understand this is not trivial.

One alternate idea that came up during internal discussion is if a config file path could be extended to allow a URL. Would this be feasible? This would work for the majority of our repositories.

[humitos]

One alternate idea that came up during internal discussion is if a config file path could be extended to allow a URL. Would this be feasible? This would work for the majority of our repositories.

I'm not 100% sold on this idea yet, but it can be something we could discuss with the core team. I'd like to evaluate more use cases that other users could bring to the conversation as well.

[humitos]

We talked about in our yesterday's meeting and we came up with some ideas for different approaches:

1. Define and standardize a readthedocs-init.sh file that will be executed immediately after the repository is clone if the file exists. This file can be used to "dynamically generate" the .readthedocs.yaml that the build system will read and parse to set up the build process.
2. Support !!python syntax on .readthedocs.yaml to allow executing Python code from inside the configuration file itself.
3. Implement global/organization environment variables and allow expansion on .readthedocs.yaml.
4. Add a new config key like build.jobs.init to allow executing a command re-generate a new .readthedocs.yaml that the build system will re-read and re-parse after executing that command.

We are not 100% convinced of any of these ideas because all of them has pros and cons. However, we would like to have more feedback from users having multiple projects and hitting this situation that could help us to decide which one will be better for your use cases.

[humitos]

I just found that MkDocs is having a similar conversation that could be useful to keep an eye on: mkdocs/mkdocs#3331