tooling: generate JSON schemas from protobuf definitions

[norbjd]

Commit Message

Generate JSON schemas from protobuf definitions. This is mostly useful to validate syntax when writing the config, using for example IDE extensions (the most well known is probably redhat's vscode-yaml one).

Additional Description

The process to write correct envoy YAML configuration files feels inefficient. As of now, there is no way to validate the syntax while editing the config - except raw YAML validation - so the process (at least for me) is basically always: 1) write config file and 2) run validate command using an Envoy Docker image (with the version I want). As a result, the developer feedback loop is slow.

This PR is an attempt to generate JSON schemas from protobuf definitions, so that they could be included in our favorite IDEs for validation. There might be other use cases I have not in mind right now that could now be possible with these.

Some examples of what it looks like after generating and using the JSON schemas (in vscode).

Autocompletion

Fields:

Values (interestingly, autocompletion for enums like the following also gives numerical values, but it works!):

Syntax validation (here, typing)

---

Note that there are already a few issues mentioning this:

• VSCode syntax extension #13078
• Add an envoyconf linting/validation tool #13233
• Add jsonschema dump of API protocols #13254

And some attempts to solve them:

• https://github.com/jcchavezs/envoy-config-schema

There is even one draft PR on Envoy:

• [WIP] tooling: Add jsonschema dump of API (...cont) #15100

However, there is no activity on these ones for at least one year (and often more) as of today.

The main reason and the main blocker for this was, I think, the fact that it was not really easy to integrate the actual plugin to generate JSON schemas from protobuf files (https://github.com/chrusty/protoc-gen-jsonschema). This is mostly because this external repository was not "bazelified". So, in order to reduce the amount of code in Envoy codebase and to simplify, I've decided to first add Bazel support to that plugin: chrusty/protoc-gen-jsonschema@master...norbjd:add-bazel, just for Envoy. If everything works here with my PR on Envoy, then I'll create the PR to merge upstream and contribute to chrusty's repository.

Once done, it's now "pretty" easy to generate JSON schemas from Envoy protobuf files. I'm putting "pretty" between quotes because I'm not a Bazel expert, and it looks like there are many conventions for Bazel in Envoy that I don't fully master. But since I had some bandwidth to work on that, I'm happy to be able to contribute! I still have a few questions, and will open a comment in this PR for everyone of them, so that Bazel masters could give me some advice. I'm also available on Slack (my username is norbjd).

Finally, there is still one remaining question about the expected output. Ideally, we'd like to have a single JSON schema file (or two maybe, one for v2 and one for v3), but the protoc plugin does not handle this, so some work may need to be done in Envoy repository to merge all generated JSON schemas together. This/these final JSON files could be exposed on Envoy website, or in a first step, just downloadable from Github releases artifacts. I guess it's something that could be done in a second step to not overload this PR and avoid scope creep.

Below you can find commands to test the feature.

1. Setup a clean environment with docker

git clone -b generate-json-schemas-from-proto --single-branch [email protected]:norbjd/envoy.git
cd envoy/
docker run --rm -it -v $PWD:/app -w /app debian:sid-slim

2. Prepare the environment

All commands from here are run inside the docker container.

apt update && apt install -y curl git gcc

# install bazelisk
curl -Lo /usr/local/bin/bazel https://github.com/bazelbuild/bazelisk/releases/download/v1.16.0/bazelisk-linux-amd64
chmod +x /usr/local/bin/bazel

# add a non-root user, required for other non-related bazel rules
adduser --disabled-password --gecos "" envoy
su envoy # connect with non-root user

3. List new rules

There are two ways to generate JSON schemas: using rules_proto_grpc or an aspect.

I don't know which one to use yet, so I've done both ways and let Envoy people decide.

# get the list of rules (using `rules_proto_grpc`)
$ bazel query 'kind("jsonschema_compile", //tools/protojsonschema:*)'
//tools/protojsonschema:envoy_config_bootstrap_v2
//tools/protojsonschema:envoy_config_bootstrap_v3

# get the list of rules (using an aspect)
bazel query 'kind("protojsonschema_rule", //tools/protojsonschema_with_aspects:*)'
//tools/protojsonschema_with_aspects:api_protojsonschema

4. Build

4.1. Using rules_proto_grpc

# example: generate the JSON schema for v3 bootstrap
$ bazel build //tools/protojsonschema:envoy_config_bootstrap_v3
...

# list generated files by the jsonschema_compile rule
$ ls -1 bazel-bin/tools/protojsonschema/envoy_config_bootstrap_v3
Admin.schema.json
Bootstrap.schema.json
ClusterManager.schema.json
CustomInlineHeader.schema.json
FatalAction.schema.json
LayeredRuntime.schema.json
Runtime.schema.json
RuntimeLayer.schema.json
Watchdog.schema.json
Watchdogs.schema.json

$ build all JSON schemas
bazel build $(bazel query 'kind("jsonschema_compile", //tools/protojsonschema:*)')
...

4.2. Using an aspect

# generate all JSON schemas
$ bazel build //tools/protojsonschema_with_aspects:api_protojsonschema
...

# list generated files by the api_protojsonschema rule (example here: v3 bootstrap)
$ ls -1 bazel-bin/external/envoy_api/envoy/config/bootstrap/v3/jsonschema
Admin.json
Bootstrap.json
ClusterManager.json
CustomInlineHeader.json
FatalAction.json
LayeredRuntime.json
Runtime.json
RuntimeLayer.json
Watchdog.json
Watchdogs.json

Risk Level

Low

Testing

I'm not really sure how to test this, other than just building the JSON schemas.

Docs Changes

/

Release Notes

Generate JSON schemas from proto files.

Platform Specific Features

/

Fixes

#13078, #13233, #13254.

[phlax]

hi @norbjd thanks for working on this

i worked on it previously and almost got it working at that time, but other priorities took over

it was quite a while ago, so i dont rem exactly, but i think it was not implemented optimally

i think probably this requires use of a bazel aspect

[phlax]

we dont want to do this

its possible that this list could be derived by using a genquery but almost certainly this should be implemented using an aspect, which allows you to traverse dependencies, and visit individual targets

[norbjd]

👍 done with the aspect, see tools/protojsonschema_with_aspects (and slight modification on tools/api_proto_plugin/plugin.bzl).

[phlax]

i would suggest a first step to just focus on getting a working protc-gen-jsonschema binary in bazel

[dio]

Cool. This is awesome.

[repokitteh-read-only]

CC @envoyproxy/dependency-shepherds: Your approval is needed for changes made to (bazel/.*repos.*\.bzl)|(bazel/dependency_imports\.bzl)|(api/bazel/.*\.bzl)|(.*/requirements\.txt)|(.*\.patch).
envoyproxy/dependency-shepherds assignee is @phlax

🐱

Caused by: #27640 was synchronize by norbjd.

see: more, trace.

[norbjd]

Hello @phlax 👋 and thanks for your useful comments!

i would suggest a first step to just focus on getting a working protc-gen-jsonschema binary in bazel

I'm not sure to understand sorry 😕

There is already a protoc-gen-jsonschema binary working in the generated binaries folder:

$ ls -1 bazel-out/*/bin/external/com_github_chrusty_protoc_gen_jsonschema/cmd/protoc-gen-jsonschema/protoc-gen-jsonschema_/protoc-gen-jsonschema
bazel-out/k8-opt-exec-2B5CBBC6/bin/external/com_github_chrusty_protoc_gen_jsonschema/cmd/protoc-gen-jsonschema/protoc-gen-jsonschema_/protoc-gen-jsonschema

# check it works by just displaying version
$(!!) -version
v1.4.0

This is the binary used to generate the JSON schemas I've shown in my PR description.

Have I misunderstood something, do you have something else in mind for "getting a working protc-gen-jsonschema binary in bazel"? Thanks again!

[phlax]

I'm not sure to understand sorry confused

could you separate out adding protoc-gen-jsonschema to its own PR please - we can then ensure everything is working with that before looking at adding an aspect

[norbjd]

Hello @phlax, done 👍 #27661 I've put main as the destination branch of this new PR, but if you want to have an intermediate branch to avoid pushing work in progress stuff to main, tell me.

About aspects, I have played a little with them, based on your work on tools/protoxform: 8f93ad2. I haven't succeed to make it work, but I feel like I'm close. The problem being that we can't really infer the output file generated by protoc-gen-jsonschema, because it depends on what messages are inside the proto file (e.g. resource.proto will generate a file ResourceAnnotation.json, some proto files with multiple messages in them are generating one file per message). I'll be happy to have your thoughts on that.

Anyway, for now, even if it's not the right solution, and as an experiment to try to better understand bazel (please bear with me, I'm still a beginner in bazel 🙏), I have considered using genquery to list the protos, I'm not really happy with the result, but at least there is now an automated way to create and maintain a list of targets in a separate file: https://github.com/envoyproxy/envoy/blob/536b9c2b7255320b850fc87e3eb1419447492fc5/tools/protojsonschema/BUILD.

[phlax]

/wait for #27661

[phlax]

so, @norbjd once #27661 lands we can figure out the next step

im also trying to think of the overall plan - im thinking the final outcome we want is that CI is automatically publishing the schemas somewhere

re use of an aspect - my niggling doubt is that im wondering if we ran protoc with the plugin against just the Bootstrap proto (and its dependencies) would that produce what we are after

i think when i tried this before i did it using an aspect, and then merged the files that it created (with various fixes/cleanups) - im wondering if that is necessary and whether this could actually be a lot simpler

[norbjd]

Hello @phlax, sorry for the late reply, I've been busy this week 🥵

I've rebased that branch and made a few changes to make //tools/protojsonschema work again. I had to use rules_proto_grpc here, but as we said in the other PR, this is probably not the right solution and we may have to resort to using aspects. I've just made this to have something temporarily working, just to see generated JSON schemas, until //tools/protojsonschema_with_aspects is fully functional and can generate schemas 👍

im wondering if we ran protoc with the plugin against just the Bootstrap proto (and its dependencies) would that produce what we are after

You're right. To take a concrete example, if we generate all JSON schemas from all existing protos, and if we take a look at the generated BootstrapConfigDump.schema.json, we can find inside that file the definition for envoy.config.cluster.v3.CircuitBreakers.Thresholds.RetryBudget:

"envoy.config.cluster.v3.CircuitBreakers.Thresholds.RetryBudget": {
    "properties": {
        "budget_percent": {
            "$ref": "#/definitions/envoy.type.v3.Percent",
            "additionalProperties": true
        },
        "min_retry_concurrency": {
            "additionalProperties": true,
            "type": "integer"
        }
    },
    "additionalProperties": true,
    "type": "object"
}

But, this exact same definition is also found in other generated JSON schemas:

• CircuitBreakers.schema.json
• Cluster.schema.json
• Clusters.schema.json
• ClusterStatus.schema.json

However, this may not be something generalizable to all protos, for example for specific filters definition that don't appear in the Bootstrap generated JSON schema.

[ravenblackx]

/wait

@phlax is on vacation at the moment so may be less responsive for a while.
Looks like there's some presubmits to be resolved in the meantime.

[phlax]

apologies @norbjd ive been a bit busy - ill look through this again as soon as i get chance

[ravenblackx]

Sorry, I didn't even notice this was assigned to me until right now. I don't think I'd be particularly helpful anyway though, I'm not familiar with IDE json-based validation, nor with bazel aspects (my earlier comment was just because I was on-call and sweeping through the things that were stuck - I guess at least I managed to help move it along a little!)

I'm going to unassign me to avoid causing confusion.

[htuch]

@phlax friendly ping on next steps for this PR.

[KBaichoo]

friendly ping @phlax

[phlax]

huge apology for delayed response - it just that this will take some time to review/test and ive been quite busy

ill try and look at this before the end of the week

/wait-any

[norbjd]

Hello 👋

Kindly bumping this PR as it has been two months without activity 😇

As a quick summary, this PR is "blocked" right now because I'm defining two ways to generate JSON schemas, and we need to decide on the best one:

1. with proto_plugin (from rules_proto_grpc)
2. with aspects: see tools/api_proto_plugin/plugin.bzl and tools/protojsonschema_with_aspects

Personally, I'm not sure which one is the best solution as (1.) is potentially impacting the existing, because we have to import rules_proto_grpc, who can create conflicts, and (2.) is modifying api_proto_plugin_impl, and I'm not that familiar with Bazel to see if my changes are ok. So I'm relying on Bazel experts here 😄

As soon as we decide on a solution, I just need to remove the files related to the other one, and it should be good to go 👍

Thank you!

[phlax]

Kindly bumping this PR as it has been two months without activity 😇

again a huge apology - entirely my fault - the delay is that this takes some testing/iteration and i have been v busy

we have a batch of releases about to go out - once these are out of the way i will priroritize this - apologies again @norbjd

[phlax]

/wait-any pending project release and further review

[chasepo]

@phlax, Do you have an timeframe on when you might have free cycles to tackle this review? I'm excited to be able to have some autocomplete support in VSCode for the yaml files, even if it's minimal at first! Thanks in advance.

[phlax]

apologies it again slipped my mind and the holiday season kicked in

no promises - but ill try and look at this next week

[alyssawilk]

@phlax ping?

[ravenblackx]

@phlax has been pinged again through back channels more aggressively, so be assured this is still in the pipeline for attention, it's just a big thing and time is hard to find.

[phlax]

@norbjd apologies for taking so long to review this

lets land this and iterate, i think its a great step forward, lgtm

[norbjd]

Thanks @phlax! Just to remind, this PR showed 2 solutions because I was really unsure of the best one:

1. with proto_plugin (from rules_proto_grpc)
2. with aspects: see tools/api_proto_plugin/plugin.bzl and tools/protojsonschema_with_aspects

I'm not sure we needed both solutions to be merged to be fair. But, if you prefer one, tell me and I'll gladly make another PR to keep only one ;)

[phlax]

thanks @norbjd i vaguely remembered that to be the case - my preference is for the aspect approach - that was kinda what delayed me - wanting to test it

[davidfiala]

Thank you for all of the work! I tried to catch up on the latest progress and how to use the artifacts of this, but I only had partial luck when trying to use it with the VSCode YAML extension.

Roughly speaking, I:

• used bazel 6.5.0
• downloaded https://github.com/envoyproxy/envoy/archive/refs/tags/v1.30.2.tar.gz
• ran bazel build tools/protojsonschema_with_aspects:api_protojsonschema
• observed artifact: bazel-bin/external/envoy_api/envoy/config/bootstrap/v3/jsonschema/Bootstrap.json
• installed https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml
• configured my project's vscode json settings to add: "yaml.schemas": { "/the/path/to/bazel-bin/..../Bootstrap.json": ["*envoy*.yaml", "*envoy*.yml"] }

When in the root of a envoy yaml file, I indeed do see that autocomplete is working and provides pop up info. 🎉

As soon as I enter a field that uses a typed_config, the autocomplete can only suggests type_url and value. It doesn't seem to understand the: typed_config: "@type": ... that is passed in.

ie:

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address: { address: 127.0.0.1, port_value: 10000 }
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          # autocomplete / validation stops working within a typed_config

Again, thank you for all the effort! I'm not sure if I'm jumping in early and we're knowingly not yet at a stage where this is supposed to work- or if I'm perhaps doing something wrong.

As an aside, it would be nice to generate these JSON artifacts as part of the release process so that others do not need to repeat the build steps.

[norbjd]

Hello @davidfiala 👋 I remember that the generated JSON schema was not perfect for some fields (namely the filters). This is because we can't easily express in the JSON schema that fields in typed_config depends on the name in the filter (e.g. if filter is envoy.filters.network.http_connection_manager, fields under typed_config will be codec_type, stat_prefix, ..., as defined here). There might be a way to do it, but it will require some changes (see below).

Anyway first, just to explain why auto-completion on filters is not working: if you look into Bootstrap.json, the filters field is defined like:

"filters": {
    "items": {
        "$ref": "#/definitions/envoy.config.cluster.v3.Filter"
    },
    "type": "array"
}

And the #/definitions/envoy.config.cluster.v3.Filter is defined (I've removed descriptions for readability) like:

"envoy.config.cluster.v3.Filter": {
    "properties": {
        "name": {
            "minLength": 1,
            "type": "string"
        },
        "typed_config": {
            "properties": {
                "type_url": {
                    "type": "string"
                },
                "value": {
                    "type": "string",
                    "format": "binary",
                    "binaryEncoding": "base64"
                }
            },
            "additionalProperties": true,
            "type": "object"
        }
    },
    "additionalProperties": true,
    "type": "object",
    "title": "[#protodoc-title: Upstream filters]\n Upstream filters apply to the connections to the upstream cluster hosts."
}

As you can see, in typed_config, we just have type_url and value (an arbitrary string) + some "additional properties" (meaning anything: this is why we don't have auto-completion). To do things properly, we would have to:

1. generate the JSON schema for filters (I don't fully remember but I think these are not generated, correct me if I'm wrong)
2. include them in Bootstrap.json
3. configure properties in envoy.config.cluster.v3.Filter with something like conditionals, to represent things like "if filter name is http_connection_manager, then properties are codec_type, stat_prefix, etc". Maybe conditionals is not the appropriate solution though, but I can't think of something else right now

Note that this is just theoretical and might not be as easy as it seems! The library used here to convert protos to JSON schema probably lacks support for this.

As an aside, it would be nice to generate these JSON artifacts as part of the release process so that others do not need to repeat the build steps.

Completely agree on this. As said in this comment, "the final outcome we want is that CI is automatically publishing the schemas somewhere". But, alas, this has not been done yet. Merging this PR was just a first step to introduce JSON schemas, so there is still a lot of things to do! Unfortunately, I'm not sure if this is a priority today.

[phlax]

hi @norbjd @davidfiala just to clarify that the name is not so important wrt extensions - its more the @type field that determines what can be set

trying to rem whether i got this working when i first tried it - i thought i had - either way, i think it should be possible if non-trivial

would be great to get this working good enough to be usable, at that point we can think about publishing somewhere