Filebeat CEL input - basic auth does not work

[a03nikki]

• Version: 8.6.2

• Operating System: MacOS Venture 13.2.1

• Discuss Forum URL:

• Steps to Reproduce:

  1. Download latest version of Filebeat, extract, and update filebeat.yml
  2. Add CEL input that points at a URL that requires username and password
     a. https://www.elastic.co/guide/en/beats/filebeat/8.6/filebeat-input-cel.html#_auth_basic_enabled
     b. x-pack/filebeat/input/cel: new input #31233
  3. Run Filebeat

     ./filebeat -v

  4. Review log files
  5. Search the code for the possible problem. It appears the config may not be passed on the request.
     a. https://github.com/elastic/beats/blob/main/x-pack/filebeat/input/cel/input.go
     b. func newClient has logic for OAuth but not basic

metricbeat.yml

filebeat.inputs:
- type: cel
  id: cel-1
  interval: 1m
  resource.url: https://api.ipify.org/?format=json
  resource.tracer.filename: logs/cel-1.trace.log
  program: |
    bytes(get(state.url).Body).as(body, {
        "events": [body.decode_json()]
    })
  enabled: true
- type: cel
  id: cel-2
  interval: 1m
  resource.url: https://REDACTED.es.us-east-2.aws.elastic-cloud.com/
  auth.basic.enabled: true
  auth.basic.user: REDACTED
  auth.basic.password: REDACTED
  auth.oauth2.enabled: false
  resource.tracer.filename: logs/cel-2.trace.log
  program: |
     bytes(get(state.url).Body).as(body, {
          "events": [body.decode_json()]
     })
  enabled: true


filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
setup.kibana:

cloud.id: REDACTED
cloud.auth: REDACTED

output.elasticsearch:

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~


logging.level: debug
logging.selectors: ["*"]

monitoring.enabled: true
monitoring.cluster_uuid: REDACTED

logs

% head -n2 cel-2.trace.log
{"log.level":"debug","@timestamp":"2023-02-17T14:25:50.343-0600","message":"HTTP request","transaction.id":"907UAHJFMP21E-1","url.original":"https://REDACTED.es.us-east-2.aws.elastic-cloud.com/","url.scheme":"https","url.path":"/","url.domain":"REDACTED.es.us-east-2.aws.elastic-cloud.com","url.port":"","url.query":"","http.request.method":"GET","user_agent.original":"","http.request.body.content":"","http.request.body.bytes":0,"http.request.mime_type":"","event.original":"GET / HTTP/1.1\r\nHost: REDACTED.es.us-east-2.aws.elastic-cloud.com\r\nUser-Agent: Go-http-client/1.1\r\nAccept-Encoding: gzip\r\n\r\n","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-02-17T14:25:50.494-0600","message":"HTTP response","transaction.id":"907UAHJFMP21E-1","http.response.status_code":401,"http.response.body.content":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","http.response.body.bytes":459,"http.response.mime_type":"application/json;charset=utf-8","event.original":"HTTP/1.1 401 Unauthorized\r\nConnection: close\r\nContent-Type: application/json;charset=utf-8\r\nDate: Fri, 17 Feb 2023 20:25:50 GMT\r\nWww-Authenticate: Basic realm=\"security\" charset=\"UTF-8\"\r\nWww-Authenticate: Bearer realm=\"security\"\r\nWww-Authenticate: ApiKey\r\nX-Cloud-Request-Id: M3_hK0hSTOCwhVTfSRbiNw\r\nX-Found-Handling-Cluster: REDACTED\r\nX-Found-Handling-Instance: instance-0000000000\r\n\r\n{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","ecs.version":"1.6.0"}

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

Thanks for finding this. The reason for the omission is in the origin of the auth code (originally from the HTTPJSON input where basic auth is handled per request and so I missed it). I need to think of a good way to thread this through to http lib, but in the meantime it is possible to work around the lack within CEL if username and password are passed into the CEL state. They can then be used to construct the request header as shown in this example. https://github.com/elastic/mito/blob/e6cd4d27d605c9a51d784926ed11cf085e59ced1/lib/http.go#L146-L164

//     request("GET", "http://www.example.com/").with({"Header":{
//         "Authorization": "Basic "+string(base64("username:password")),
//     }})
//
//     will return:
//
//     {
//         "Close": false,
//         "ContentLength": 0,
//         "Header": {
//             "Authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ="
//         },
//         "Host": "www.example.com",
//         "Method": "GET",
//         "Proto": "HTTP/1.1",
//         "ProtoMajor": 1,
//         "ProtoMinor": 1,
//         "URL": "http://www.example.com/"
//     }

The request can then be used with do_request. This is more involved and I will sort out a fix for the base case.

[efd6]

I have a good path forward to making this work as documented. Depends on the mito/lib PR tagged above.

diff --git a/x-pack/filebeat/input/cel/config.go b/x-pack/filebeat/input/cel/config.go
index 69f3512f13..13d9112b14 100644
--- a/x-pack/filebeat/input/cel/config.go
+++ b/x-pack/filebeat/input/cel/config.go
@@ -71,7 +71,7 @@ func (c config) Validate() error {
        if len(c.Regexps) != 0 {
                patterns = map[string]*regexp.Regexp{".": nil}
        }
-       _, err = newProgram(context.Background(), c.Program, root, client, nil, patterns)
+       _, err = newProgram(context.Background(), c.Program, root, client, nil, nil, patterns)
        if err != nil {
                return fmt.Errorf("failed to check program: %w", err)
        }
diff --git a/x-pack/filebeat/input/cel/input.go b/x-pack/filebeat/input/cel/input.go
index b2a7acd41a..b9f4a2e868 100644
--- a/x-pack/filebeat/input/cel/input.go
+++ b/x-pack/filebeat/input/cel/input.go
@@ -122,7 +122,14 @@ func (input) run(env v2.Context, src *source, cursor map[string]interface{}, pub
                return err
        }
 
-       prg, err := newProgram(ctx, cfg.Program, root, client, limiter, patterns)
+       var auth *lib.BasicAuth
+       if cfg.Auth.Basic.isEnabled() {
+               auth = &lib.BasicAuth{
+                       Username: cfg.Auth.Basic.User,
+                       Password: cfg.Auth.Basic.Password,
+               }
+       }
+       prg, err := newProgram(ctx, cfg.Program, root, client, limiter, auth, patterns)
        if err != nil {
                return err
        }
@@ -808,7 +815,7 @@ var (
        }
 )
 
-func newProgram(ctx context.Context, src, root string, client *http.Client, limiter *rate.Limiter, patterns map[string]*regexp.Regexp) (cel.Program, error) {
+func newProgram(ctx context.Context, src, root string, client *http.Client, limiter *rate.Limiter, auth *lib.BasicAuth, patterns map[string]*regexp.Regexp) (cel.Program, error) {
        opts := []cel.EnvOption{
                cel.Declarations(decls.NewVar(root, decls.Dyn)),
                lib.Collections(),
@@ -825,7 +832,7 @@ func newProgram(ctx context.Context, src, root string, client *http.Client, limi
                }),
        }
        if client != nil {
-               opts = append(opts, lib.HTTPWithContext(ctx, client, limiter))
+               opts = append(opts, lib.HTTPWithContext(ctx, client, limiter, auth))
        }
        if len(patterns) != 0 {
                opts = append(opts, lib.Regexp(patterns))