fix(oas): improve examples validation

[P0lip]

Fixes #901

Checklist

• Tests added / updated
• Docs added / updated

Does this PR introduce a breaking change?

• Yes
• No

[nulltoken]

@P0lip or maybe just use lodash isObjectLike?

[P0lip]

Yeah, I considered doing it, but hm, tree-shaking feels shaky 😆
Let me see how it impacts the size of the bundled fn.
It's the difference is negligible, I'll go for isObjectLike

[philsturgeon]

These rules have always been hell, due to the incredibly confusing nature of all the different types of examples that can show up anywhere in OAS2, OAS3, and JSON Schema.

It honestly might need custom functions.

Using https://github.com/stoplightio/studio/issues/381 as a reference to all the bajillion different combinations of rule, can we think about how we'd implement what combination of examples to handle:

Schema Objects (for OAS2, OAS3, JSON Schema) when there could be x-example or example or examples or x-examples depending on whatever type of thing is there, and the value is always a bare value, or an array of bare values.

Then there's Media Types, Parameters, etc which are sometimes "Schema like" but not, or might actually have a "Schema" inside them...

No one rule could do everything, and we keep getting bug reports when we miss a certain specific combination, so whatever we do needs to work for everything.

P.S No need to validate externalValue.

[philsturgeon]

Related to my other comment: schema examples can be multiple: true but do not have a value key.

{
  "title" : "Match anything",
  "description" : "This is a schema that matches anything.",
  "default" : "Default value",
  "examples" : [
    "Anything",
    4035
  ]
}

[P0lip]

@philsturgeon I added oas2-valid-example rule.

[philsturgeon]

Excellent! I'm going to try this out on https://github.com/philsturgeon/examples-example and see how far we get. I'd suggest we use these as a basis for test harness.

Once we've got the ones which are meant to fail to actually fail, we can try them with $ref variants too.

[philsturgeon]

Amazing work here, running this branch on from https://github.com/philsturgeon/examples-examples (the oas3.yaml) I am seeing these errors:

 145:30  error  oas3-valid-content-schema-example    `example` property type should be boolean
 149:30  error  oas3-valid-content-schema-example    `example` property should match format `date-time`
 153:30  error  oas3-valid-content-schema-example    `example` property should match format `url`
 176:20  error  oas3-valid-oas-parameter-example     `example` property should be equal to one of the allowed values: foo, bar
 192:22  error  oas3-valid-parameter-schema-example  `example` property should be equal to one of the allowed values: foo, bar
 204:22  error  oas3-valid-oas-parameter-example     `value` property type should be string
 206:21  error  oas3-valid-oas-parameter-example     `value` property type should be string
 269:25  error  oas3-valid-oas-content-example       `value` property should have required property `id`
 289:25  error  oas3-valid-oas-content-example       `value` property should have required property `user`

The first 5 are in v5.5 but the last 4 are only in this branch. The errors are not wildly useful as they are, but they are catching onto the fact that there are problems in the examples.

204:22 error oas3-valid-oas-parameter-example value property type should be string
206:21 error oas3-valid-oas-parameter-example value property type should be string

        - name: multiple-examples
          description: Valid to its schema
          in: query
          schema: 
            type: string
            enum: [foo, bar]
          examples: 
            the-good:
              value: foo
            the-bad:
              value: 123                       # line 204: it's noticed this isn't a string!
            the-ugly:
              value: [an, array]            # line 206: it's noticed this isnt a string either

It's also noticed some response examples using the examples keywords:

        "500":
          description: Fail, because the example is nonsense
          content:
            application/json:
              schema:
                properties:
                  id:
                    type: integer
                  name:
                    type: string
                  completed:
                    type: boolean
                  completed_at:
                    type: string
                    format: date-time
                required:
                  - id
                  - name
                  - completed
              examples:
                response:
                  value:
                    completed: Not a boolean
                    completed_at: Not a date

So that's cool.

It's still missing this:

      "500":
        description: Fail, because the example is nonsense
        content:
          application/json:
            schema:
              properties:
                id:
                  type: integer
                name:
                  type: string
                completed:
                  type: boolean
                completed_at:
                  type: string
                  format: date-time
              required:
                - id
                - name
                - completed
              # notice how im in the schema object, not next to it
              example:
                id: 1
                completed: Not a boolean
                completed_at: Not a date

I'll have a look at OAS2 too.

[philsturgeon]

For oas2.yaml in that repo I'm seeing these two errors:

 176:30    error  oas2-valid-response-example  `application/json` property should have required property `name`
 193:30    error  oas2-valid-response-example  `application/json` property should have required property `user`

That's two more than the latest release!

Similarly to oas3 feedback, if a response example is an object that sits inside a schema, it doesnt seem to get picked up. This example should fail, but doesn't:

        '500':
          description: 'Fail, because the example is nonsense'
          schema:
            properties: 
              id:
                type: integer
              name:
                type: string
              completed: 
                type: boolean
              completed_at:
                type: string
                format: date-time
            required:
              - id
              - name
              - completed
            # notice how im in the schema object, not next to it
            example:
              id: 1
              completed: Not a boolean
              completed_at: 'Not a date'

OAS2 "property examples" should also fail, but currently do not. These do work in OAS3.

        '501':
          description: 'Fail, because the property example is nonsense'
          schema:
            properties: 
              some-bool:
                type: boolean
                example: not a bool
              some-date:
                type: string
                format: date-time
                example: not a date
              some-url:
                type: string
                format: url
                example: not a URL

In OAS2, parameter examples don't seem to be failing as I suppose we aren't looking for x-example (the only way to make an example for a parameter in OAS2).

  /param-examples:
    get:
      operationId: parameter-example
      description: when parameter examples are inline (no $ref involved)
      tags:
        - parameter-examples
      parameters:
        - name: single-example-good
          description: Valid to its schema
          in: query
          type: string
          enum: [foo, bar]
          x-example: foo
        
        - name: single-example-bad
          description: Not valid to its schema
          in: query
          type: string
          enum: [foo, bar]
          x-example: not "foo" or "bar" so this is bad

[P0lip]

@philsturgeon I think this should be ready for another round.

[philsturgeon]

Amazing work, you nailed every single example in these documents.

I gave this a go on the GitHub OpenAPI in sample-specs, and I'm seeing this output:

They've actually got some errors in their doc, and I'll talk to them about that, but why is this first one double reporting for a single error? Is this a $ref issue?

[P0lip]

@philsturgeon ah, seems like the same rule targets the same portion of the spec.
I'll tweak them.

[P0lip]

@philsturgeon how do you feel about merging some of the rules?
Since given might be an array now, we do not need to have plenty of rules such as oas2-valid-response-schema-example or oas2-valid-parameter-example. What do you think?
As this is a potential breaking change, we might postpone it until 6.0

[philsturgeon]

Yeah I thought we could probably get away with some rule merges so long as we can get the right function options into place. I let seems like different targets will need different function options, but if you can get the tests to pass with the changed rules then... go ahead!

Maybe we can do the minimal change now and merge the rules as a follow up in 6.0, like you said.

[P0lip]

Maybe we can do the minimal change now and merge the rules as a follow up in 6.0, like you said.

I'd need to drop a rule or two and merge them into some existing one.
I'll create a follow-up PR, so that 5.5 can be released without any breaking changes included.