Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parse and resolve schema $refs w/out validating #136

Closed
neilpa opened this issue Aug 3, 2023 · 6 comments
Closed

Parse and resolve schema $refs w/out validating #136

neilpa opened this issue Aug 3, 2023 · 6 comments

Comments

@neilpa
Copy link

neilpa commented Aug 3, 2023

Is it possible to get the fully resolved schema with all the $refs replaced with the actual definitions (assuming no recursion). I've got some fairly complex schemas spread across a few dozen local files that get rolled into one schema that I'd like to get a unified view of. I've been digging into the code and AFAICT the only time $refs get resolved is during validation.

@davishmcclurg
Copy link
Owner

Hi @neilpa—that's not currently possible, but I'd like to add it as a feature after 2.0.0 is released.

It looks like the spec calls the generated schemas compound documents and the resolution process bundling.

@neilpa-inv
Copy link

neilpa-inv commented Aug 4, 2023

Thanks for the links. I was able to easily hack something together that pre-processes the $ref values and inlines the remote schemas or local $defs before handing it off to JSONSchemer. Definitely not robust, but we already have a custom ref_resolver so I was able to leverage that.

@unicornzero
Copy link

I'm also interested in pre-validation ref resolution, but haven't dug into that yet. (Too many things to research, too little time!) @neilpa-inv I'd be curious to see what you hacked together as a starting point, if you don't mind sharing.

@neilpa-inv
Copy link

@unicornzero Unfortunately I can't share the actual code. Roughly speaking, it looks like this with the replacement step omitted since that depends on the various $refs you need to handle. (I'm also very much not a Ruby person so there may be a more idiomatic way to do this).

q = [ JSON.parse("path/to/root/schema.json") ]
while !q.empty?
  obj = q.shift
  if obj.is_a? Hash
    if obj.key? "$ref"
      ref = obj["$ref"]
      obj.delete_if { true }
      # add values to `obj` from the the resolved remote $ref or local $def
      q.unshift(obj)
    else
      q.concat(obj.values)
    end
  elsif obj.is_a? Array
    q.concat(obj)
  end
end

@unicornzero
Copy link

Thanks for sharing what works for you! I'll refer back to this when I'm next digging into this part of things.

davishmcclurg added a commit that referenced this issue Aug 19, 2023
This allows schemas with remote refs to be combined into a single
portable schema by resolving refs and embedding them in `$defs`. I
believe I matched the process described in the [specification][0],
though I'm not entirely sure it'll work in all cases (especially id/ref
fragments).

The implementation walks through the schema, finds any ref-like keywords
(identified by `ref_uri`/`ref_schema`), and dumps the referenced schema
into `$defs`. It always uses the root schema so that the ref URI
fragment still works for applying a JSON pointer (or
location-independent identifier) as expected. I had trouble thinking
through all the possibilities here, but it seems to work ok in testing.

Embedded schemas may have their $id values overwritten, but I believe
it's necessary because schemas may not be referenced by a matching $id
(see different-id-ref-string.json).

Related:

- #136

[0]: https://json-schema.org/draft/2020-12/json-schema-core.html#section-9.3
davishmcclurg added a commit that referenced this issue Aug 19, 2023
Features:

- Draft 2020-12 support
- Draft 2019-09 support
- Output formats
- Annotations
- OpenAPI 3.1 schema support
- OpenAPI 3.0 schema support
- `insert_property_defaults` in conditional subschemas
- Error messages
- Non-string schema and data keys
- Schema bundling

See individual commits for more details.

Closes:

- #27
- #44
- #55
- #91
- #94
- #116
- #123
- #136
@davishmcclurg davishmcclurg mentioned this issue Aug 19, 2023
@davishmcclurg
Copy link
Owner

I added bundling support to the next branch for release in 2.0.0. If you want to try it out, it currently works like this:

schema = {
  '$id' => 'http://example.com/schema',
  'allOf' => [
    { '$ref' => 'schema/one' },
    { '$ref' => 'schema/two' }
  ]
}
refs = {
  URI('http://example.com/schema/one') => {
    'type' => 'integer'
  },
  URI('http://example.com/schema/two') => {
    'minimum' => 11
  }
}
schemer = JSONSchemer.schema(schema, :ref_resolver => refs.to_proc)

schemer.bundle
# => {"$id"=>"http://example.com/schema",
#     "allOf"=>[{"$ref"=>"schema/one"}, {"$ref"=>"schema/two"}],
#     "$schema"=>"https://json-schema.org/draft/2020-12/schema",
#     "$defs"=>
#      {"http://example.com/schema/one"=>{"type"=>"integer", "$id"=>"http://example.com/schema/one", "$schema"=>"https://json-schema.org/draft/2020-12/schema"},
#       "http://example.com/schema/two"=>{"minimum"=>11, "$id"=>"http://example.com/schema/two", "$schema"=>"https://json-schema.org/draft/2020-12/schema"}}}

davishmcclurg added a commit that referenced this issue Aug 20, 2023
This allows schemas with remote refs to be combined into a single
portable schema by resolving refs and embedding them in `$defs`. I
believe I matched the process described in the [specification][0],
though I'm not entirely sure it'll work in all cases (especially id/ref
fragments).

The implementation walks through the schema, finds any ref-like keywords
(identified by `ref_uri`/`ref_schema`), and dumps the referenced schema
into `$defs`. It always uses the root schema so that the ref URI
fragment still works for applying a JSON pointer (or
location-independent identifier) as expected. I had trouble thinking
through all the possibilities here, but it seems to work ok in testing.

Embedded schemas may have their $id values overwritten, but I believe
it's necessary because schemas may not be referenced by a matching $id
(see different-id-ref-string.json).

Related:

- #136

[0]: https://json-schema.org/draft/2020-12/json-schema-core.html#section-9.3
davishmcclurg added a commit that referenced this issue Aug 20, 2023
Features:

- Draft 2020-12 support
- Draft 2019-09 support
- Output formats
- Annotations
- OpenAPI 3.1 schema support
- OpenAPI 3.0 schema support
- `insert_property_defaults` in conditional subschemas
- Error messages
- Non-string schema and data keys
- Schema bundling

See individual commits for more details.

Closes:

- #27
- #44
- #55
- #91
- #94
- #116
- #123
- #136
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants