Autogenerated reference docs for Helm Chart

[mik-laj]

Hello,

We have JSON schema definition for values.yaml - values.schema.json This can us generate user-friendly reference documentation. Currently the documentation is not updated very often and does not describe all fields. I don't know a tool that can do that, but I imagine it could be displayed as a tree. There may be a ready tool to generate documentation for Helm file or generic documentation generation tools for JSON Schema, but I haven't checked it yet.

This will allow us to keep the reference documentation up-to-date and to make it more user-friendly.

Best regards,
Kamil Breguła

[mik-laj]

@flvndh Can you help with it?

[flvndh]

It would be a pleasure 😁

[flvndh]

@mik-laj How should this script be triggered? Also, how do you see the output being rendered ?

[mik-laj]

I am flexible. One solution i see is generating this file with every release and publishing on airflow.apache.org. See: #1115

Do you have another idea? I'd love to talk about it.

[flvndh]

At first glance I would have generated it the same way as the documentation for airflow.cfg:

• Add a new .rst file in airflow/docs using Jinja
• Provide the Jinja context in airflow/docs/conf.py (see this)

Is this what you were thinking about?

[mik-laj]

@flvndh Do we need a Sphinx? I think we can use custom Python script together with Jinja It will be much easier to set up and maintain. As you can see in #1115, I would like each of our products to have separate documentation and therefore a separate documentation building process. If you wanted to use Sphinx, you would have to create a full Sphinx configuration for Helm only. And then don't use any Sphinx features, just write your own extension.

[flvndh]

Sorry @mik-laj but I'm a bit confused by your reference. License headers are handled by pre-commit now, right ?

That make me think that we could write a custom hook that would update the chart README whenever values.yaml or values.schema.json is modified. What's your opinion?

[mik-laj]

@flvndh Yes. We use pre-commit checks to update README.md files Example below:

airflow/.pre-commit-config.yaml

Lines 235 to 240 in ebd7150

	- id: update-breeze-file
	name: Update output of breeze command in BREEZE.rst
	entry: "./scripts/ci/pre_commit/pre_commit_breeze_cmd_line.sh"
	language: system
	files: ^BREEZE.rst$|^breeze$|^breeze-complete$
	pass_filenames: false

Sphinx does not generate MD files, but HTML files, so your original proposal assumed the workflow would be as follows.

1. Users run Sphinx
2. Sphinx load Jinja extension
3. Sphinx load values.schema.json
4. Jinja extension renders RST page based on values.schema.json file
5. Sphinx render RST as HTML
6. User has an HTML file and can publish them.

Since the rest of the documentation is in README.md. And I don't see the need to move it, only we will have one file with reference documentation.

I think this can be simplified and we can write a simple Python script that will load a template with HTML code, load the configuration, and generate reference documentation.

It will look a bit like the code below.

TPL_PATH = ....

TEMPLATE_LOADER = jinja2.FileSystemLoader(searchpath=TPL_PATH)

TEMPLATE_ENV = jinja2.Environment(
    loader=TEMPLATE_LOADER, undefined=jinja2.StrictUndefined, trim_blocks=True, lstrip_blocks=True
)
TEMPLATE_CACHES: Dict[str, Any] = {}

def render_template(template_name: str, *args, **kwargs) -> str:
    """Render Jinja template"""
    if template_name not in TEMPLATE_CACHES:
        template = TEMPLATE_ENV.get_template(template_name)
        TEMPLATE_CACHES[template_name] = template
    content: str = TEMPLATE_CACHES[template_name].render(*args, **kwargs)
    return content

def load_schema(schema_filepath: str) -> str:
	with open(schema_filepath) as json_file
		return json.load(json_file)

TEMPLATE_PATH = "./helm-refernece-docs.html"
SCHEMA_PATH = "../values.schema.json"
OUTPUT_PATH = "../dist/reference-docs.html"

with file(output_filepath, "w") as output_file:
	schema = load_schema(SCHEMA_PATH)
	output_file.wirte(render_template(TEMPLATE_NAME, schema=schemaa))

print("Refenrece docs generated to {output_filepath}")

While writing this code, I found one cool tool that can help us too - j2-cli
Then generating documentation may look much easier.

j2 refernece-docs.j2 ../values.schema.json

You also ask if it should be a markdown or HTML. I don't have strong requirements on this, but I think HTML will be much more flexible. We can achieve a much better quality and user-experience than in pure markdown. On the other hand, if we use markdown, our documentation will always be updated thanks to pre-commit checks. I prefer HTML a little more, but I think if we use j2-cli it will be easy to support both formats.

[mik-laj]

@flvndh I have prepared a small POC:
Code: https://github.com/PolideaInternal/airflow/tree/helm-docs-ref
Preview: https://thundering-color.surge.sh/
If you want, you can be inspired by it or develop your own solution. What do you think about it?

It would be fantastic if it looked similar to Redoc.

Source: https://airflow.readthedocs.io/en/latest/stable-rest-api-ref.html

[mik-laj]

@flvndh Do you need help? For now, we have to maintain the documentation manually, but it would be great if it could be done automatically. Especially that recently we have a lot of new valuable contributions to Helm Chart, and without documentation users will not benefit from it.

[flvndh]

Sorry @mik-laj I was quite busy these days but would be able to come with something soon.

[mik-laj]

We took a step forward! I talked to the authors of the ArtifactsHub project about my idea and they liked it so much that they immediately implemented it and deployed it to the server.

You can take a look at it live in the Artifact Hub package: https://artifacthub.io/packages/helm/artifact-hub/artifact-hub

Now all we have to do is add our Helm Chart to the repository to make the documentation available to everyone.