Custom HTTP queries for webhooks

[hooksie1]

I think it would be an awesome feature to allow custom headers in webhooks. The main thought is this would let us use systems that require some kind of authentication for the webhooks.

Talked about this in discord, tagging @vviers.

[vviers]

we plan on working on this, cc @CamilleLegeron

Quoting @fflorent :

Thinking of it, it may be interesting to allow defining the methods, headers and shapes of the request body in columns so their values can be computed through formulas.

[paulfitz]

Neat! A workaround is passing requests through an external service like cloudflare workers to rewrite them, but that is a pain to setup and then annoying to maintain.

[paulfitz]

With @CamilleLegeron working on webhooks in #832, I wonder if we could sneak a field for this into the migration she'll be introducing. Just a generic options column would do, we wouldn't have to commit to any representation yet. We typically have an options columns on tables for non-relational stuff that shouldn't require every Grist document everywhere in the world to be migrated when something is added, but there isn't one yet on the _grist_Triggers table. Doing it now would save another migration later.

[CamilleLegeron]

What about security if there is an authorization secret ?
Inside webhook settings page maybe add an input for secret and another for none secret ? And that way we can encrypte the secret and set both in options column

[CamilleLegeron]

Thinking about what headers could be great to custom, I'm only thinking of the authorization parameter. How about adding only Authorization input in the webhooks params and see later if other needs arise?
Wdyt @paulfitz ?

[paulfitz]

Thinking about what headers could be great to custom, I'm only thinking of the authorization parameter. How about adding only Authorization input in the webhooks params and see later if other needs arise?
Wdyt @paulfitz ?

There's nothing very special to Grist about webhooks. Could be worth looking at what similar apps offer, to see if there is any consensus?

What about security if there is an authorization secret ?
Inside webhook settings page maybe add an input for secret and another for none secret ? And that way we can encrypt the secret and set both in options column

The security question is interesting. The webhook URL itself may have a sensitive key in it, which is part of why it is not stored directly in the document but in a secrets table in the home database. This makes some things more awkward and some things simpler., and makes adding more encryption in the future easier. It could make sense to treat material for Authorization header in the same way, at least partially.

I wonder if this needs actually two features:

• A way to store and retrieve secrets associated with a particular document (the secrets table is intended for that purpose)
• A way to include a secret in a header, for example with some kind of syntax that could later extend to pulling information from the payload itself or elsewhere? Like Authorization: {secrets.AUTH_SECRET} or something. Maybe this is too ambitious. But it would mean the header configuration itself wouldn't have to be treated as a secret.

[fflorent]

I tend to think it be more interesting to change this issue title so it tracks three similar features that will probably be implemented altogether (otherwise please mention your disagreement about that), which are the ability to customize the following things for webhooks:

• the method;
• the body;
• the headers;

(as stated in an above comment)

[nicolas-imbert]

Hey,

Following a quick talk with @fflorent and some users, I just wanted to add my 2 cents on the potential use cases for this feature.

1. Send an email when there is a new record or updated record (via tools like Brevo or Mailchimp)
2. Send a notification when new record or updated record (via tools like Mattermost or Teams)

And probably a lot more to come !
Looking forward to it, thanks for the job done.

[fflorent]

Regarding the implementation, what about allowing configuring a column in which it is expected to return these items thanks to a formula, like:

import json
return {
  'method': 'PATCH', # Defaults to Post
  'headers': { # Headers to append to the one sent by the webhooks (like the headers)
    'X-Specific-Header': $someColumn,
    'X-Common-Header': 'Common value for all records',
  },
  'body': json.dumps({ # the body, defaults to nothing
    'whatever': $someOtherColumn
  }),
  'queryParams': 'blah=' + $queryParams # Defaults to empty string.
  # Other stuffs?
}

(The queryParam property came to my mind when writing this formula)

What do you think?

[fflorent]

We would need to clarify the User Story, prevent security issues and have a design mockup for this issue.

[vviers]

Tracking progress here :

• add Authorisation header in webhooks (Add authorization header in webhooks stored in secrets table #941)
• allow user to specify the shape of a header sent via webhooks
• allow user to specify the shape of a payload sent via webhooks
• allow user to specify query params sent via webhooks
• allow user to specify the HTTP verb used by webhooks

@hooksie1 could you please rename this issue to the more generic "Custom HTTP queries for webhooks"

[jordigh]

Regarding the implementation, what about allowing configuring a column in which it is expected to return these items thanks to a formula, like:

I like this implementation. But I also think the default payload (array of rows that triggered the webhook) is a good idea. I think the default payload should be available to the Python formula, maybe a special variable such as $TRIGGERING_ROWS or similar.

Since you wrote this idea, @CamilleLegeron has already implemented an authorization header, which I think is a good initial approach, but I hope we can keep working towards @fflorent's idea to provide a flexible Python formula that can define all parameters of the request, with defaults provided if fields are missing. My rationale is that we need programmatic flexibility anyway to generate the payload body in general. The user needs the ability to write arbitrary logic to generate the webhook payload.