Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal for Swagger Async Responses and WebHooks #716

Closed
RobPhippen opened this issue Jun 16, 2016 · 13 comments
Closed

Proposal for Swagger Async Responses and WebHooks #716

RobPhippen opened this issue Jun 16, 2016 · 13 comments

Comments

@RobPhippen
Copy link

RobPhippen commented Jun 16, 2016

Swagger Asynchronous Reponses and WebHooks

Background

This document provides an outline of the proposed swagger format for documenting a webhook subscription and callback interface.

Background

Many SaaS systems provide some kind of event interface as core to their behaviour.
There are many approaches to implementing this. No single approach is as prevalent as JSON/HTTP APIs have become for service implementation however webhooks is significtly more common than any other single approach.

This document concentrates on extending Swagger to cope with the WebHooks pattern.

WebHooks

Webhooks are a convention of use of normal Web interfaces, typically using JSON/HTTP,
in which an event subscriber can register a callback URL with an event publisher.
When an event occurs,
the event publisher invokes an HTTP method on that URL (Typically POST) to send the information related to the event.

WebHooks in real life

Webhooks are not standardized nor formally specified, but are in common use in a substiantial
number of systems.

The lack of standardisation means that there is variation in each of the following
areas;

  • Whether or not a subscription API is made available - many systems instead require
    registration via an administrative UI
  • If a subscription API is available, its precise form varies between different systems
  • Some systems effectively require a different callback URL to be registered for each event stream,
    while others support subscription to multiple event streams with a single callback URL

The following conventions are commonly observed;

  • When an event occurs, where the publisher supports subscription to multiple different event streams on the same URL
    it is common for them to provide a parameter, or a data field within a parameter, as part of the
    POSTed message that specifies which event has occurred.
  • When an event occurs, almost all systems support the use of HTTP POST to convey the event information.
    Some mandate the use of HTTP POST, while others allow other HTTP methods to be selected as an option

Objective of the proposal

This objective of this proposal is to allow the subscription and event publication interactions
involved in the WebHooks convention to be minimally modeled via Swagger, such that a Swagger-enabled SDK could;

  • Understand event structures: Provide an abstraction of each event that could occur once subcription has occurred, reducing the amount of code that a programmer has to author to process the events.
  • Discriminate between events: discriminate between events of different types that may occur so that the appropriate program logic can be invoked
  • Subcription/Publication association: (by allowing the event set+discriminator to be associated with a subscription API)
    allow a programmer to discover the events that can occur once they have registered
  • Documentation: extend the documentation benefits of swagger to events.

Not covered by the proposal

This proposal does not cover abstraction of the variation encountered in different
implementations of the subscription API such that an SDK could standardize this phase.

Specific Proposal

x-async-responses

New tags: x-async-responses is array of event definitions. Each event definition contains the following elements;

  • event-id: a logical name for the event stream
  • description: a text description of the event stream
  • parameters
    • This allows the signature of the HTTP POST performed when an event occurs in the source system to be modelled
    • With the exception of eventDiscriminator it is identical to a normal Swagger parameter definition
    • In the webhooks convention, the event payload is contained on the body of the HTTP method invoked
  • eventDiscriminator:
    • Parameter: parameterName
    • This introduced to cope with cases where the webhooks callback URL can receive more than one type of event and the event type is conveyed in the header or query perameters. In these cases, the polymorphic Swagger discriminator cannot be used. The Parameter field specifies which parameter is used to discriminate between different events that may occur.
    • If eventDiscriminator is present, then the parameter named has a value equal to the event-id when that event occurs
    • If eventDiscriminator is absent, then event types are discriminated using the Swagger discriminator, as specified for the schema of the body parameter
  • Usage in the context of a subscription API
    • If x-async-responses is included or referred to as an element within an operation definition,
      as shown in this example, this is an indication
      that the events defined may occur as a result of invoking that operation.
  • Usage standalone
    • If x-async-responses appears at the top level,
      this indicates that the subscription happens via some unspecified mechanism.
    • It is common for systems that support WebHooks to provide
      a UI that allows the subscription to be performed, for example

Discriminator in a header: Swagger YAML snippet

We have chosen to illustrate example using github because it has a subcription API.

Please refer to the Github documentation on
'create a hook'
for further information on the API definition provided by Github for webhooks.

Please note that all elements beginning with x-github or x-hub are part of the Github API
specification, and do not form part of this proposal.
Finally, the Swagger below is not comprehensive - for example, not all objects are fully modeled.

paths:
  /repos/owner/repo/hooks:
    post:
      operationId: 'CreateHook'
      summary: Create a webhook in GITHUB
      description: |
        Call this API to create a new WebHook in Github
      parameters:
        - name: subscription
          in: body
          description: Details to subscribe for a callback on a given set of events
          required: true
          schema:
            $ref: '#/definitions/Subscription'
      tags:
        - Webhook
      responses:
        '202':
          description: Details of the subscription created to the webhook
          schema:
            $ref: '#/definitions/SubscriptionResponse'
      x-async-responses:
        - eventId: commentStream
          description: Comment event stream
          eventDiscriminator:
            parameter: X-GitHub-Event
          parameters:
            - name: comment
              in: body
              required: true
              schema:
                $ref: '#/definitions/comment'
            - name: X-GitHub-Event
              in: header
              type: string
            - name: X-GitHub-Delivery
              in: header
              type: string
            - name: X-Hub-Signature
              in: header
              type: string
        - event-id: deployment
          description: deployment event stream
          eventDiscriminator:
            parameter: X-GitHub-Event
          parameters:
            - name: deployment
              in: body
              required: true
              schema:
                $ref: '#/definitions/deployment'
            - name: X-GitHub-Event
              in: header
              type: string
            - name: X-GitHub-Delivery
              in: header
              type: string
            - name: X-Hub-Signature
              in: header
              type: string      

Discriminator in the message Body

In this example, the eventDiscriminator is absent, and defaults to exploiting the discriminator defined in the schema for the body parameter.

x-async-responses:
  - event-id: AnimalStream
    description: animal event stream
    parameters:
      - name: animal
        in: body
        required: true
        schema:
          $ref: '#/definitions/Pet'

definitions:
  Pet:
    type: object
    discriminator: petType
    properties:
      name:
        type: string
      petType:
        type: string
    required:
    - name
    - petType
  Cat:
    description: A representation of a cat
    allOf:
    - $ref: '#/definitions/Pet'
    - type: object
      properties:
        huntingSkill:
          type: string
          description: The measured skill for hunting
          default: lazy
          enum:
          - clueless
          - lazy
          - adventurous
          - aggressive
      required:
      - huntingSkill
  Dog:
    description: A representation of a dog
    allOf:
    - $ref: '#/definitions/Pet'
    - type: object
      properties:
        packSize:
          type: integer
          format: int32
          description: the size of the pack the dog is from
          default: 0
          minimum: 0
      required:
      - packSize

Wider objectives not covered in this document

  • Support semantic tagging for the subscription APIs
    • In order to allow an SDK to support abstraction of the subscription mechanism
  • Support Business annotations
    • events, actions, and data can have business annotations, such as;
      • Display Name
      • Description
      • Semantic Tags
  • Support other event propagation protocols
    • Support swagger representation of other event propagation protocols, such as
      • Kafka
      • etc

Complete Sample swagger for GitHub

swagger: '2.0'
info:
  title: Webhooks example - for Github
  description: |
  version: 1.0.0
host: example.com:443
schemes:
  - https
basePath: /
produces:
  - application/json

paths:
  /repos/owner/repo/hooks:
    post:
      operationId: 'CreateHook'
      summary: Create a webhook in GITHUB
      description: |
        Call this API to create a new WebHook in Github
      parameters:
        - name: subscription
          in: body
          description: Details to subscribe for a callback on a given set of events
          required: true
          schema:
            $ref: '#/definitions/Subscription'
      tags:
        - Webhook
      responses:
        '202':
          description: Details of the subscription created to the webhook
          schema:
            $ref: '#/definitions/SubscriptionResponse'
      x-async-responses:
        - event-id: commentStream
          description: Comment event stream
          eventDiscriminator:
            in: X-GitHub-Event
          parameters:
            - name: comment
              in: body
              required: true
              schema:
                $ref: '#/definitions/CommentEventPayload'
            - name: X-GitHub-Event
              in: header
              type: string
            - name: X-GitHub-Delivery
              in: header
              type: string
            - name: X-Hub-Signature
              in: header
              type: string
        - event-id: deployment
          description: deployment event stream
          parameters:
            - name: deployment
              in: body
              required: true
              schema:
                $ref: '#/definitions/DeploymentEventPayload'
            - name: X-GitHub-Event
              in: header
              type: string
              event-discriminator-value: 'deployment'
            - name: X-GitHub-Delivery
              in: header
              type: string
            - name: X-Hub-Signature
              in: header
              type: string      
definitions:
  Subscription:
    type: object
    properties:
      name:
        type: string
      config:
        type: object
        properties:
          url:
            type: string
          content_type:
            type: string
      events:
        type: array
        items:
          type: string
      active:
        type: boolean

  SubscriptionResponse:
    type: object
    properties:
      id:
        type: number
      url:
        type: string
      test_url:
        type: string
      ping_url:
        type: string
      name:
        type: string
      events:
        type: array
        items:
          type: string
      active:
        type: boolean
      config:
        type: object
      updated_at:
        type: string
      created_at:
        type: string


  CommentEventPayload:
    type: object
    properties:
      action:
        type: string
      comment:
        $ref: '#/definitions/Comment'
      repository:
        $ref: '#/definitions/Repository'
      sender:
        $ref: '#/definitions/Sender'

  DeploymentEventPayload:
    type: object
    properties:
      deployment:
        $ref: '#/definitions/Deployment'
      repository:
        $ref: '#/definitions/Repository'
      sender:
        $ref: '#/definitions/Sender'  

  Comment:
    type: object

  Repository:
    type: object

  Sender:
    type: object

  Deployment:
    type: object


@wparad
Copy link

wparad commented Jun 16, 2016

Events (Pub/Sub) is just one possible use of the Open API, why would the API include a preferred implementation? Assuming this could be done, let's investigate why:

Understand event structures: Provide an abstraction of each event that could occur once subscription has occurred, reducing the amount of code that a programmer has to author to process the events.

Isn't this already covered by Definitions?

Discriminate between events: discriminate between events of different types that may occur so that the appropriate program logic can be invoked

This in effect forces a coupling of the service you are expecting to implement the proposal of this issue with the client which calls the service API. Rather than building the logic to handle events in the client (or service API code generator, if you are using one), this proposal would seek to force those implementation details on the service.

Subscription/Publication association: (by allowing the event set+discriminator to be associated with a subscription API) allow a programmer to discover the events that can occur once they have registered

The same could be done easily with GET /event-types.

Documentation: extend the documentation benefits of swagger to events.

I propose this can be solved without any changes, using a mediatype of application/json+event (creation pending) and benefit all code generators and well as service owners who implement handling this directly instead of only one documentation tool.

There is an argument for the Asynchronicity of API which sends you messages, that's a little weird when it comes to REST, i.e. normally those would be a part of your API and not the service's. (right? It's the other service acting as the client, and your client acting as the service is this case) Alternatively, we could easily model the service API to include a GET call with a definition that resembles a envelope object:

{
    eventMetadata: object,
    event: object
}

The only part not accounted for is that this API will actually send this response irrelevant of a GET call. Okay so we are saying that the service will know to send out data in this fashion? You register with the service and it remembers your API's event information, and later it will send out events to that location. Creating an async property to represent that nature could be done, however...

Doesn't that violate the very nature of REST, i.e. the service sending out state in which it had to remember? If events really do violate REST then they can't be included in the Open API specification.

What could be included? Well.... it seems as if an RFC for a webhook type is in order, after a quick search I didn't find one. The service API in which you are a client of would just specify that RFC as the mediatype for its calls, and clients would automatically understand.

@ePaul
Copy link
Contributor

ePaul commented Jun 16, 2016

So basically you are defining the request part of an operation (i.e. parameters, description), but without a response, which needs to be implemented by the API consumer, and somehow link it to the provider's operation as an "async response". But it doesn't really say which URL submitted by the consumer will be used then for the callback, and also duplicates a lot of the structure in

I can only point to my interfaces proposal in #577 (and a prototype spec change in #650), which, while originally meant for the HATEOAS use case, also can be used for this use case. I guess discriminating between event types in the same URI by parameters would be done the same way a solution for #164 or #146 is done, so here is just an example with two different URIs for the two events. (Of course, the usual discriminator-in-object solution works here too.)

definitions:
   Subscription:
     type: object
     properties:
       commentCallback:
         type: string
         description:
           The callback to which the events will be sent.
         interface:
           $ref: "#/interfaces/CommentCallback"
       deploymentCallback:
         type: string
         description:
           The callback to which the events will be sent.
         interface:
           $ref: "#/interfaces/DeploymentCallback"
paths:
  /repos/owner/repo/hooks:
    post:
      operationId: 'CreateHook'
      summary: Create a webhook in GITHUB
      description: |
        Call this API to create a new WebHook in Github
      parameters:
        - name: subscription
          in: body
          description: Details to subscribe for a callback on a given set of events
          required: true
          schema:
            $ref: '#/definitions/Subscription'
      tags:
        - Webhook
      responses:
        '202':
          description: Details of the subscription created to the webhook
          schema:
            $ref: '#/definitions/SubscriptionResponse'
interfaces:
  CommentCallback:
    post:
      description:
        Will be called whenever a new comment is posted.
      parameters:
        - name: comment
          in: body
          required: true
          schema:
            $ref: '#/definitions/comment'
        - name: X-GitHub-Event
          in: header
          type: string
        - name: X-GitHub-Delivery
          in: header
          type: string
        - name: X-Hub-Signature
          in: header
          type: string
      responses:
        default:
          description: The response is ignored by the hook.
  DeploymentCallback:
    post:
      description:
        Will be called whenever a deployment event happens.
      parameters:
        - name: deployment
          in: body
          required: true
          schema:
            $ref: '#/definitions/deployment'
        - name: X-GitHub-Event
          in: header
          type: string
        - name: X-GitHub-Delivery
          in: header
          type: string
        - name: X-Hub-Signature
          in: header
          type: string
      responses:
        default:
          description: The response is ignored by the hook.

(I guess we would also need to define responses, even if they are ignored by the hooks caller.)

@RobPhippen
Copy link
Author

Hi @ePaul I find your thought interesting. A key characteristic of the 'callback' API is that The server fundamentally cannot know the base of the URL it calls back on until registration has happened. So the proposal deliberately leaves that 'unbound' in the static Swagger that describes the event emitter. I think your interfaces thought achieves that.

@RobPhippen
Copy link
Author

RobPhippen commented Jul 12, 2016

It was agreed on the OAI TDC on Friday 17th June 2016 that this proposal would be split into three items;

@fehguy
Copy link
Contributor

fehguy commented Jul 28, 2016

From the OAI TDC call, here are some notes:

/subscribe:
  post:
    host:
      # i forgot the syntax :-/
    parameters:
      - in: formData
        name: url
        type: string
        format: url
    responses:
      200:
        description: you're registered!
    callbacks:
      # callbacks are sent to the url provided
      # this could also be a query param, etc.
      url: '$request.url'
      schema:
        type: object
        properties:
          a:
            type: string

/subscribe:
  post:
    schema:
      type: object
      properties:
        url:
          type: string
          format: url
    responses:
      200:
        description: you're registered!
    callbacks:
      # callbacks are sent to the url provided
      # this could also be a query param, etc.
      url: '$request.url'
      schema:
        type: object
        properties:
          a:
            type: string

/subscribe:
  post:
    schema:
      type: object
      properties:
        url:
          type: string
          format: url
    responses:
      200:
        description: you're registered!
    callbacks:
      $ref: '#/components/callbacks/somethingNeat'

components:
  callbacks:
    somethingNeat:
      externalDocs:
        url: http://www.my.server.com/registerCallback
        description: set up this callback here ^^ :-S
      url: '$request.url'
      schema:
        type: object
        properties:
          a:
            type: string

# or

/subscribe:
  post:
    parameters:
      - in: query

    responses:
      200:
        description: you're registered!
    callbacks:
      # callbacks are sent to the url provided
      # this could also be a query param, etc.
      url: '$request.url'
      schema:
        type: object
        properties:
          a:
            type: string

example

client:
  POST /subscribe
  body:
{
  url: http://my.server.com/callme/maybe
}    
server:
  # at some point in time
  POST http://my.server.com/callme/maybe
  body:
{
  "a": "it works"
}

@leslie-wang
Copy link

This is useful feature. Is someone still actively working on it?

@fehguy
Copy link
Contributor

fehguy commented Jan 31, 2017

Yes please look at the 3.0 draft spec, which merges #763

@leslie-wang
Copy link

Thanks for updating. can you please share when 3.0 is going to publish?

@webron
Copy link
Member

webron commented Jan 31, 2017

@leslie-wang
Copy link

cool! Looking forward to practicing.
BTW, guess swagger editor (other tools) need add support too.

@webron
Copy link
Member

webron commented Jan 31, 2017

All in due time.

@darrelmiller
Copy link
Member

Included in V3 https://github.com/OAI/OpenAPI-Specification/blob/OpenAPI.next/versions/3.0.md#callbacks-object

@CGamesPlay
Copy link

Hello, in the initial proposal here:

  • Usage standalone
    • If x-async-responses appears at the top level, this indicates that the subscription happens via some unspecified mechanism.
    • It is common for systems that support WebHooks to provide
      a UI that allows the subscription to be performed, for example

This appears to have gotten lost in the various shuffles. Is there any current or planned method to document a callback that is not created from a specific path?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants