Build: implement build.commands override of build commands

[agjohnson]

In #8190 we started discussing a path forward for making our build processes more flexible for users.

The first addition to our build process is #9016, which allows for pre/post build step command execution that is configurable by the user. The build.jobs addition should support many more use cases for projects already. That is, projects could execute a command before Read the Docs executes sphinx-build, etc.

The concept behind build.commands is that projects could override the actual commands we execute with their own commands -- so override sphinx-build entirely, not just execute commands before/after. This would be most useful to replace sphinx-build with a build tool of your choice, however there are a number of implementations that expect Sphinx exclusively that we'd need to change.

I'm trying to summarize a chain of several design discussions, but what would be left for this portion of the work is:

• Gather some feedback on build.commands and it's usage
• Build: implement build output contract #9088
• TBD but altering our build time injection points and search indexing? Build: support build time injected logic via generic HTML injection #9063
• Alter our current build commands to be independently overridden

Refs #1083
Requires #9063

[humitos]

I just want to be clear here that if we are following the proposal that I've done in #8190 we cannot implement build.commands without defining and implementing the build contract (metadata.yaml) first. Otherwise, Read the Docs won't be able to know where to look the output files nor how to integrate our features with the resulting HTML.

[agjohnson]

Sure, my description is by no means definitive. We just need to capture what you have planned, and communicate that for everyone. Feel free to expand on the description here or make new issues.

[agjohnson]

One point of feedback that I had here -- and this relates to the metadata.yaml conversation as well -- was that I'm not sure I'd need more control or complexity over our build environment.

I used build.jobs to hack up support for Pelican, and really only felt that I needed some concept of a generic backend instead of the sphinx backend. I'm maybe describing a simpler version of build.commands + metadata.yaml concepts, but the intent is the same.

A user might have a configuration file like:

build:
  jobs:
    pre_build:
      - pelican content
  backend:
    generic:
      output:
        html: ./output/

The generic builder doesn't do much of anything, except for perhaps moving files for syncing?

To support Pelican, this was all I really found myself wanting. I had to hack up Sphinx to be no-op step, and had to hack up the paths maintained by the Sphinx backend, build.jobs gave me everything else I wanted.

[humitos]

really only felt that I needed some concept of a generic backend instead of the sphinx backend. I'm maybe describing a simpler version of build.commands + metadata.yaml concepts, but the intent is the same.

I want to avoid the concept of backends directly integrated with Read the Docs as much as possible. I don't want to integrate these backends in the application and that's why I separated this into these two concepts:

• build.jobs: we do all the things and users can write hooks in between our regular commands. This will cover Sphinx / MkDocs being integrated and maintained by ourselves.
• build.commands: the user is responsible for all the build process and just needs to respect a contract. This will cover any other backed, integrated and supported by the community.

Mixing the two concepts makes it really confusing for the user because this distinction becomes gray. Besides, making the two of them work together will make the integration in our application a lot more complex; which is what I wanted to avoid originally.

Your example with the UX that I have in mind (without the application knowing anything about backends) would be:

# .readthedocs.yaml
build:
  jobs:
    commands:
      - pip install pelican
      - pelican build

# metadata.yaml
output:
  html: ./output/

Note that I want the metadata explicitly being in a separate file: https://github.com/readthedocs/readthedocs.org/pull/8190/files#r636080468

[agjohnson]

Yeah, I'm caught up on the plan so far. However, I'm saying our plan might be a bit too rigid at the moment. I see a slightly more defined path with an additional step before implementing build commands.

We have discussed build job hooks as a way to provide users with build flexibility, and build command overrides as a way to provide communities with the ability to support the documentation tools they maintain. I still agree with this plan as a long term approach, but think we would benefit greatly from a short term approach too. There is the strong possibility that with build.jobs and support for artifact upload, build.commands is just redundant.

I suppose no matter what, the metadata yaml is still required if we accept built tarballs for projects. The UX of combining the build contract file with our configuration file is interesting but a separate conversation. For this use case, I'm not even convinced user configuration of the output files is necessary, but agree we'll want a file at some point.

Anyways, until we get build command overrides, users have no ability to support arbitrary documentation tools without heinous hacks to Sphinx. The piece I'm describing above is a generic builder that collects files but does not execute commands. This seems like a technical piece required for build commands overrides either way, so is only complimentary to what we've defined so for for work.

With the ability for users to determine the files uploaded to storage, we can fairly immediately support arbitrary tooling. This is also where we have to say "this is experimental and most features probably don't work yet, but give us feedback".

My primary concern is this: we have a large number of unanswered questions about support for arbitrary tooling and need some way to ease in. Without some middle step, it seems we have to take a huge gamble all at once:

• First we wait, but it's not really clear what we're waiting on here at the moment -- seeing what users do with build.jobs? How do we know users need build.commands? How do we know if communities will come to us?
• Implement metadata contract
• Implement generic builder to opt out of mkdocs/sphinx builder class
• Implement build command overrides
• Implement generic support for build-time injected content
• Finally, we can rally support for community maintenance of third party builders and hope it takes off.

That's a big gamble and a lot of work. If we expand the generic builder a little in a middle step, and make it possible to support build file collection with a generic builder class, the steps then become:

• Implement generic builder to opt out of mkdocs/sphinx builder class
• Get users using build jobs and generic builder to use their custom doc tooling
• See what is working and what is commonly being used, poll communities to see if they would formally support a builder class? At this point we're also figuring out how to support ad injection in a generic way, generic support for a flyout, and perhaps generic search indexing? There are lots of questions at this level.
• If things look positive, add build commands and metadata contract on top of build jobs to expand flexibility further.

A generic build class can start off without configuration even -- we can dictate to the user what hard coded files we copy. The only major difference in the plan seems to be the ability to opt in to a non-sphinx/mkdocs build class in our configuration. Even if we mark this as experimental, at least we have the ability to ease into a world where we support output besides mkdocs/sphinx.

[agjohnson]

Note that I want the metadata explicitly being in a separate file: #8190 (files)

Can we break this out to an issue finally? That chain of design docs is overwhelming.

edit: I've made a TBD placeholder at #9088

[agjohnson]

Setting this as need more information as we're not sure if/what we need. Still on the backlog though.

.. Dammit no response bot. Whatever, design decision.

[humitos]

@agjohnson

Honestly, I don't like the idea of implementing a hybrid or something "in between" build.jobs and build.commands. That will be a lot of things to maintain for our team and we will be responsible for many issues that won't work in our integration because we run some default commands in build.jobs the user wasn't expected to be run.

I originally proposed build.jobs and build.commands to simplify the number of possible combinations and also to delegate the responsibility to the user if something fails on build.commands. When using build.commands you are by yourself, we will just upload the output/ directory. That's the message I want to communicate.

If we expand the generic builder a little in a middle step, and make it possible to support build file collection with a generic builder class, the steps then become:

Here you are basically describing build.commands without any feature integration, which is something that I thought you said you didn't want. If you are OK with this, I'd prefer our middle plan to be exactly that, actually. "Use build.commands, do whatever you want, and we will upload the output/ directory to be served" (note this is the most basic version of the metadata.yaml without requiring a file --output/ directory is hardcoded and can't be changed by the user). This does not require any integration with our existing features, does not require parsing HTML or anything similar, does not require a lot of work in the config file and it won't create a half-baked feature integrated with something that's currently working fine and well defined as build.job is.

Also, note the implementation of this specific use case of build.commands will require a small amount of work, since it only requires adding support for build.commands in the config file and only running those commands. It does not involve new concepts like "generic backend" and similar. We can treat this as experimental as you mentioned already and force requiring the existence of a metadata.yaml once we are ready for the integrations via HTML parsing.

Besides, as we are not too good at deprecating things, and we finally keep maintaining it forever, I'd prefer to not implement something "in between", as I said before.

By implementing this basic build.commands case, we will immediately support a lot of new use cases, bringing more users to the platform. We can expand this basic build.commands case with more Read the Docs integrations as we move forward. However, from start, it will have a lot of features already supported. Like versioning, automated rules, redirects, environment variables, etc. but it won't integrate with ads, search, and flyout.

As an example, your use case will be just:

build:
  os: ubuntu-22.04
  commands:
    - pip install pelican
    - pelican build
    # in case pelican does not output files in the hardcoded ./output/ directory Read the Docs expects
    - mv ./pelican-custom-output-folder/ ./output/

The resulting ./output/ folder will be uploaded and served without ads, search, or flyout.

[ericholscher]

The resulting ./output/ folder will be uploaded and served without ads, search, or flyout.

It seems like we can still do HTML parsing of the output folder, without any additional configuration information? For example, we could fail the build if the HTML doesn't have an <div id=insert-ad-here />, for a trivial example. It would also be reasonably easy to add whatever HTML we find into a search index. Additional tuning would require additional information of course, but the basic implementation isn't hard.

I tend to agree with Manuel that I don't see a lot of value in the backend config option as currently presented. I don't fully understand it, but I don't really see what value it is adding. It seems like just another way of defining the metadata in metadata.yml? Eventually we'll want a way to pass information from the build to RTD, which could also be in a metadata.yml file or in .readthedocs.yaml, but I think that metadata.yml is what we've agreed on in the past as it adds a lot more flexibility (eg. HTML upload).

It feels like we're mostly having a conversation around convention or configuration:

• Convention: We expect all output in the output directory.
• Configuration: Tell us where the output is, and we'll use that it.

I think both are fine. I think implementing build.commands with a convention of output/ is a reasonable first step. We can then add metadata.yml as needed for RTD integration?

I would argue though that we're getting ahead of ourselves, and I'd like to put this conversation on hold for 1-2 months, as we learn more about what people are doing with build.jobs, as that will strongly influence our decisions.

[agjohnson]

Let me rephrase this, because I'm not trying to focus on the config file, and it seems most of this discussion is on that piece.

I understand what we've proposed for build.commands and the discussions we've had on the topic, and to clarify again, I don't disagree with the implementation.

But the important part of my point above is that we also don't yet know if build.commands is required yet. The flexibility offered with build.jobs is already really high, and we've talked about supporting HTML upload to support complex builds (this overlaps with the what we think build.commands supports). So it's not clear where build.commands really lives yet, and support for arbitrary tooling is meanwhile blocked on this.

So, I was describing an alternate order of operations. Get a feature to support build tools in front of users early, even in experimental state, and see what users do with it. I think experiments are really important here, given the complexity of the change we're describing.

After that, we definitely need first class support for generic HTML output, we definitely need a build contract file eventually, but build.commands is still a big unknown to me.

The piece I was describing was a generic builder. Maybe it has configuration, but that is a secondary conversation. The builder backend:

• Doesn't execute commands like Sphinx/Mkdocs builder backends do
• It still collects files at the end, from static paths to start seems fine
• It gives users an immediate way to use build.jobs to use arbitrary build tools
• Is needed if we implement build.commands. We need a build backend that doesn't execute commands.

This is the step that I'm talking about changing. We implement what would eventually be the generic builder that build.commands would use, and let users use it with build.jobs. It seems like we can get away without offering configuration here.

There is one extraneous piece required to get this going though, and that is a way to opt into the generic builder. I described this as build[backend[generic]], but perhaps this is where we need to discuss options for moving feature flags to config options -- a way for users to opt into temporary/experimental features without us supporting these long term.

It does not involve new concepts like "generic backend" and similar.

It seems to me we need a generic builder either way, thinking through the technical changes. I doesn't seem like you're proposing reusing the Sphinx/Mkdocs builder class for build.commands.

[agjohnson]

Also, it's worth considering that we have some dependent steps.

• We need to know if users will come to us with arbitrary build tools before we take on the work to build generic HTML versions of embed injection, search indexing, flyouts, and ads
• We need to know if users will come to us and hit limitation with build.jobs before implementing build.commands
• We also need to know if users are better served with HTML upload before we decide if we need build.commands

So the order of operations seems it should really be something closer to:

• Allow experimental support of arbitrary doc tools. Add a generic builder backend class
• Allow experimental support for HTML upload
• Implement a build contract or decide if convention is enough
• Work on first class support for custom HTML by making generic versions of flyout/search/injection/etc.
• Determine if we still need build.commands

And most explicitly, I'm talking about taking on the first step in our short term roadmap and still pausing for a bit. We need time to accumulate information, and getting a feature in front of users and seeing what build tools are actually used is important data.

Perhaps the compromise is implementing build.commands very soon, but as an experimental feature. We don't push for marketing or support it yet. This seems okay, my main concerns are costs that are duplicated with both build.commands and build.jobs -- implementation, maintenance, support costs, and marketing costs. Consider the work we're discussing with build.jobs and marketing it, we'll have to do that again with build.commands.

It feels like we're mostly having a conversation around convention or configuration:

I was originally describing configuration, but convention indeed seems like a better option for at least what I'm describing. I'm happy to also discuss that more in #9088 as an option.

[agjohnson]

Anyways, I think we'd benefit from a follow up meeting here, and I've voiced that we'd benefit from closing up the open design doc PRs on this topic. The state of this work is a bit difficult to track as our plans mostly haven't made it out of draft PRs and meeting notes.

Some of the unknowns, and things we haven't settled on, for the project that I have are:

• What data are we waiting on right now?
  • Define this so it's not ambiguous.
• What should we be monitoring and checking in on for build.jobs usage?
  • Abusive command patterns and abusive metrics (like new project cpu usage maybe?) is high on my list
  • Users doing things that confirm our thoughts on usage -- hacking out of Sphinx builds for example
• Can we implement build.commands in a minimal way -- experimental feature that could go away, not marketing it much yet, etc
  • Can we do this soon?
• What we need to trigger the next steps of our plan
  • Generic HTML integration support, and a plan for easing into this work (there's a lot)
    • What is the order of implementation?
  • When do we test HTML upload?
    • What do we need to implement and what do we need to decide on this plan. This relates to build.commands as it's basically a third implementation of executing build commands.

Suppose there are more here, feel free to append.

[ericholscher]

Strong 👍 on a meeting, as it feels like we're talking past each other a decent bit. I think we're all mostly agreeing, and just proposing different ways to get to similar outcomes.

[agjohnson]

Yup, I'm not comparing outcomes here, they are the same effective outcome to the user. I'm only describing a difference in time frame, effort we spend, and when we take work on. We haven't talked about this, among other topics, in our planning and I think it's worth discussing.

[humitos]

I'm 👍🏼 on a meeting. I do see the path to move forward pretty clear regarding build.jobs and build.commands. Its separation from a UX perspective but also on the technical side. Actually, this was part of the original research and planning work I've done months ago when I shared these ideas with both of you. This makes me think that 1) I'm missing some important pieces that you are saying and I'm not able to see them yet or 2) I'm not doing a great job explaining myself 😄 . In any case, a meeting will definitely help in both scenarios.

That said, I wrote some comments that may help to explain a little more my position on this. Here they are,

@ericholscher

It seems like we can still do HTML parsing of the output folder, without any additional configuration information? For example, we could fail the build if the HTML doesn't have an <div id=insert-ad-here />, for a trivial example. It would also be reasonably easy to add whatever HTML we find into a search index. Additional tuning would require additional information of course, but the basic implementation isn't hard.

This could be a good initial compromise that it's compatible with the path forward we already have in mind for the contract. I'm not opposed to it. However, this requires a little more work than without this initial integration as Anthony was suggesting which could give us a pretty quick experimental POC.

I think both are fine. I think implementing build.commands with a convention of output/ is a reasonable first step. We can then add metadata.yml as needed for RTD integration?

This makes sense to me.

@agjohnson

So, I was describing an alternate order of operations. Get a feature to support build tools in front of users early, even in experimental state, and see what users do with it.

I'd like to see this happening but without introducing a new concept like "backends", and if we can implement it as the first step of build.commands as I mentioned in my previous comment. In that case, 👍🏼

This is the step that I'm talking about changing. We implement what would eventually be the generic builder that build.commands would use, and let users use it with build.jobs.

This is exactly what I described in my previous comment. You can get exactly what you want using my proposed solution. There is no need to combine build.jobs with a new concept of "generic builder". The "generic build" you are describing is build.commands already. That was my original plan when I wrote the initial design document.

On what I'm describing here, note that by using build.commands in your config file you are opting-in for the "generic builder" you are describing. There is no need to mix a backend config key/concept with the current build.jobs functionality.

I doesn't seem like you're proposing reusing the Sphinx/Mkdocs builder class for build.commands.

Right. I'm not proposing reusing the Sphinx/Mkdocs builder classes for build.commands. We won't have a Python class with hardcoded commands when using build.commands since we are not going to execute any command on behalf of the user. The user is responsible for running the correct commands to end up with the content they want to host under ./output folder. Read the Docs won't do anything to build their docs, just run the commands they specify in build.commands. Basically,

1. clone the repository
2. execute all the commands on build.commands
3. upload the ./output directory to S3

In the next iteration, once we have the contract defined, Read the Docs will parse the HTML output generated by the commands defined by the user and integrate that output with our features.

[agjohnson]

This makes me think that 1) I'm missing some important pieces that you are saying and I'm not able to see them yet or 2) I'm not doing a great job explaining myself

I understand your points, but a meeting does probably help the most here. I know the plan so far, how the plan intended to solve this, and understand the conversations we already have had around this implementation. And to clarify, I'm not questioning the original design.

I'm describing adding more granularity to our plan, and breaking out earlier implementation of the backend builder class it seems we're both describing.

We aren't sure if we need build.commands yet, but our plan for starting a beta test for supporting arbitrary tooling will be blocked this and several other steps. Meanwhile, this use case is possible with build.jobs, but the sphinx/mkdocs builders get in the way.

I proposed exposing this via build.jobs so we don't have to worry yet about the UX, marketing, and support issues that worry us with a build.commands implementation.

Right. I'm not proposing reusing the Sphinx/Mkdocs builder classes for build.commands.

It seems we're talking about the same piece then, a builder class to avoid sphinx/mkdoc command execution, but still collect files post-build. I'm just describing exposing this to users via build.jobs and build.commands.

[humitos]

I'm describing adding more granularity to our plan, and breaking out earlier implementation of the backend builder class it seems we're both describing.

This works for me. We can implement a simple initial version of build.commands to allow people to quickly experiment with different backends without too much technical effort from our side. Promote it as beta / experimental and do not spend a lot of time doing marketing around it.

This approach will quickly give us a good understanding of what people want, how they are using this to build docs with currently unsupported tools, etc. This data will help us to build better integrations with them and create a contract that supports them all in the future.

I'm just describing exposing this to users via build.jobs and build.commands

This is my main concern, here. I'm -1 on exposing this via build.jobs. This breaks the original idea of the plan and the whole separation concept that I had in mind and we have been talking about. I see our current plan introducing a new different product (build.commands) that's easy to communicate as something separated:

• magic/old builder¹: don't require even a conf.py file and execute all the default commands
• regular builder: sphinx/mkdocs structure with a .readthedocs.yaml file that executes all default commands plus hooks via build.jobs
• custom builder: executes only commands defined by the user via build.commands and uploads the resulting output folder

This vision, to me, has some good benefits by making a clear distinction between each of them. The user is able to immediately understand what's the solution that fits best for their own use case depending on how they build their docs. Also, this separation gives Read the Docs team the liberty to work on better support on "regular builder" without mixing any type of compatibility with "custom builder". It's also easier to communicate that "Read the Docs can't be responsible for any command executed at build.commands because we don't have control of that" reducing the number of support requests.

We could "allow people to override default commands via build.jobs" --which I understand is what you are suggesting-- and something we talked about before. However, I think that idea is more confusing as UX than helpful at this point and more complex to implement. For example: if the user overrides build.install and does not install readthedocs-sphinx-ext all the integrations done by this extension will be broken; even if the build succeeds. I'm not expecting broken integrations sometimes when using build.jobs but we are expecting them to be broken to some degree on the beta/experimental "custom builder".

As I said before, this separation also helps us to see "regular builder" as the current product that integrates everything and has been working for +10 years, and "custom builder" as a new product where we can have some freedom to experiment with without breaking our main, well-tested, and successful product.

Footnotes

1. potentially deprecated at some point in the future and removed from our platform completely ↩

[ericholscher]

• magic/old builder1: don't require even a conf.py file and execute all the default commands
• regular builder: sphinx/mkdocs structure with a .readthedocs.yaml file that executes all default commands plus hooks via build.jobs
• custom builder: executes only commands defined by the user via build.commands and uploads the resulting output folder

I love this overview of the builders, and this is something we need to explain in our docs in some way to users. It's a great way to help users understand which of our "builder products" they want.

[agjohnson]

We covered a few of these points in our meeting today. To summarize:

Use cases

We have several project profile use cases we're actually talking about here. To build on the point above about our builder classes, those project profiles are:

• Novice: Uses magic/old builder. I think we are still planning to deprecate this builder, upgrade these users to regular RTD users
• Regular: using basic features of our config file, perhaps including some customization using build.jobs even
• Advanced: projects that are likely (but not limited to) advanced Sphinx usage and can't quite do what they need with our config file -- customize sphinx command, pip install command, maybe override build command eventually. This is a gap in our plan that we will return to. We're expecting these users to be expert users for now, and they can manage a build.commands instead
• Expert: uses build.commands, doing something highly custom, wants us to parse HTML, on their own when things go sideways.

Our goals

Our first goal is to enable experimental support for build.commands to unlock this usage for users. However, this is under the user contract that using this feature breaks a lot of RTD features. We'll use this period to see what users need or want, and see what users actually do with the feature. We'll use this to guide our plan for future work.

Our next goal after this is to work towards first class support of HTML parsing and injection. We'll chip away here, as there is a lot of re-engineering work here -- search indexing, API injection, ads injection.

Other notes

• Implementing build.commands is one or two sprints
• Build.commands is going to be experimental, we implement this early
  • Big experimental flag wherever we discuss build.commands
  • We communicate how to use the configuration options, and what will likely be broken for users
  • We aren't yet aiming for polish here, and aren't yet focusing on marketing
  • Our build output contract is convention based, doesn't require a contract file
  • We aren't initially injecting ads, but it should be communicated that this will be part of the final feature
  • It's probably wise to implement a builder backend that could be reused by build.jobs in the future. We may decide to port implementation to build.jobs to cover the advanced project profile eventually
• While we are observing projects, we have some implementations we can be maturing and discussing more. These points are not required for an experimental phase.
  • Search indexing and configuration of indexing
  • Injection of ads
  • New API for flyout menu
  • Injection of API for flyout menu?
    • Cloudflare workers? Nginx? At build time? This is a later decision
  • Build output contract
  • UX hurdles to a polished build.commands implementation
    • How to handle PDF/EPUB output, other steps that we expect are skipped?
    • Confusing UX that most of our config file doesn't do anything now

[humitos]

The intial phase of this issue is already implemented in #9150

Implementing build.commands is one or two sprints
Build.commands is going to be experimental, we implement this early

We are ready to review it and I think it could be deployed soon as well. I think it covers our main goals for exploration and it could give us the feedback we need while keep discussing the other aspects of the more polished feature.

[humitos]

Injection of API for flyout menu?
Cloudflare workers? Nginx? At build time? This is a later decision

I just wanted to mention that I'd rather use NGINX or build time for this because using Cloudflare to inject them will make local development not work. Also, at first sight, it seems better to inject them at build time over NGINX because that's what we have been doing for years and we know it works already. Using NGINX may bring a problem with the versions of those files: how do we distinguish between a new build that needs the newer version of the js and an old project that was built 5 years ago and requires the js from that date?

[agjohnson]

Just noting that local development of Cloudflare workers is possible:

https://developers.cloudflare.com/workers/learning/debugging-workers/

I'd also vote against Nginx, we have been moving logic out of Nginx for a while now.

Build time injection does make sense so far though, and is sort of a way of versioning the injection. But a dynamic injection option does also leave some doors open for us to force changes without rebuilding the versions, if there are strong reasons to do this.