[Filebeat] Add OAuth support to the httpjson input

[andrewkroh]

As user of the Filebeat httpjson input, I would like to be able to pull data from a REST API that requires authentication via OAuth.

An example would be collecting logs from GSuite. For that an OAuth token is required.

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[marc-gr]

To start with, implementing the client credentials flow seems to be a nice choice, since it is the one allowing server-to-server authorization with no client intervention.

For this the config needs to be extended with something like:

filebeat.inputs:
- type: httpjson
  interval: 10s
  oauth2:
    token_url: http://localhost:8282/access/token/request
    client_id: dummy-client-id
    client_secret: dummy-client-secret
  url: http://localhost:8282/foobar

There are some providers, that we might want to handle but do not offer support for this flow, like Gsuite, which instead of the client_id and client_secret provides a credentials file. To handle these specific cases we might want to have a specific config object inside the oauth2, for example:

- type: httpjson
  interval: 10s
  oauth2:
    google:
        credentials_file: /path/to/file
  url: http://localhost:8282/foobar

or

- type: httpjson
  interval: 10s
  oauth2:
    provider: google
    credentials_file: /path/to/file
  url: http://localhost:8282/foobar

And do different checks based on the provider. An example of how google auth works with the credentials file can be found in the googlepubsub input

Other specific provider we might want to support is AzureAD, for example.

Also allowing for an optional scopes option to let users define specific scopes would be nice.

filebeat.inputs:
- type: httpjson
  interval: 10s
  oauth2:
    token_url: http://localhost:8282/access/token/request
    client_id: dummy-client-id
    client_secret: dummy-client-secret
    scopes:
    - scope1
    - scope2
  url: http://localhost:8282/foobar

Is there any other consideration you think might be good for the initial implementation?

cc @P1llus

[P1llus]

To start with, implementing the client credentials flow seems to be a nice choice, since it is the one allowing server-to-server authorization with no client intervention.

For this the config needs to be extended with something like:

filebeat.inputs:
- type: httpjson
  interval: 10s
  oauth2:
    token_url: http://localhost:8282/access/token/request
    client_id: dummy-client-id
    client_secret: dummy-client-secret
  url: http://localhost:8282/foobar

There are some providers, that we might want to handle but do not offer support for this flow, like Gsuite, which instead of the client_id and client_secret provides a credentials file. To handle these specific cases we might want to have a specific config object inside the oauth2, for example:

- type: httpjson
  interval: 10s
  oauth2:
    google:
        credentials_file: /path/to/file
  url: http://localhost:8282/foobar

or

- type: httpjson
  interval: 10s
  oauth2:
    provider: google
    credentials_file: /path/to/file
  url: http://localhost:8282/foobar

And do different checks based on the provider. An example of how google auth works with the credentials file can be found in the googlepubsub input

Other specific provider we might want to support is AzureAD, for example.

Also allowing for an optional scopes option to let users define specific scopes would be nice.

filebeat.inputs:
- type: httpjson
  interval: 10s
  oauth2:
    token_url: http://localhost:8282/access/token/request
    client_id: dummy-client-id
    client_secret: dummy-client-secret
    scopes:
    - scope1
    - scope2
  url: http://localhost:8282/foobar

Is there any other consideration you think might be good for the initial implementation?

cc @P1llus

I think this sounds nice, I think the second choice looked the best personally:

- type: httpjson
  interval: 10s
  oauth2:
    provider: google
    credentials_file: /path/to/file
  url: http://localhost:8282/foobar

If we are planning to use the provider list from the oauth2 lib (don't know if this is updated?), then we might be able to add the tokenURL for the user and all they would need is minimum provider, client-id, secret and URL.

From what we saw, when authentication as an app towards AzureAD we could not define scope, as it was defined when generating the ID and secret.

Maybe scope should be optional and if tokenurl is not set we should try to retrieve based on provider?

[marc-gr]

If we are planning to use the provider list from the oauth2 lib (don't know if this is updated?), then we might be able to add the tokenURL for the user and all they would need is minimum provider, client-id, secret and URL.

That is a good point, there are many providers that seem not super relevant for our use case though. Maybe doing a specific list of the ones supported would be good enough? I can think of AzureAD/microsoft and Google for a starter, since they provide extra functionality besides just the endpoints, maybe other interesting ones such as Amazon would be nice. WDYT? (the list here https://github.com/golang/oauth2/blob/master/endpoints/endpoints.go and https://github.com/golang/oauth2)

Maybe scope should be optional and if tokenurl is not set we should try to retrieve based on provider?

That sounds good 👍

[tonymeehan]

This is awesome!!!!! How cool is it to see this PR!