Add filebeat receiver to otel mode

[leehinman]

What does this PR do?

Adds filebeat receiver as an otel receiver.

Why is it important?

This is the first beat receiver, and this will allow us to test the feasibility of running beats as otel receivers.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

should be 0 user impact of existing configurations

How to test this PR locally

Use something like the following as otel.yml, and run elastic-agent otel --config otel.yml
This will run the beat receiver, which will produce one event.
The event should be written to the path you specify in the file/no_rotation section.

receivers:
  filebeatreceiver:
    filebeat:
      inputs:
        - type: benchmark
          enabled: true
          count: 1
          message: "The quick brown fox jumped over the lazy dog"
    output:
      otelconsumer:
    logging:
      level: info
      selectors:
        - '*'
    path:
      home: /tmp/<state dir>
exporters:
  file/no_rotation:
    path: /tmp/<output file>
service:
  pipelines:
    logs:
      receivers: [filebeatreceiver]
      exporters: [file/norotation]

Related issues

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 3489_beat_receiver2 upstream/3489_beat_receiver2
git merge upstream/main
git push upstream 3489_beat_receiver2

[blakerouse]

Seems we where doing double work on this one #5666

You got the integration test added before me, so I am going to close mine in favor of yours.

[blakerouse]

Does this one need to be added as well https://github.com/elastic/beats/blob/main/go.mod#L418

What about this fsnotify https://github.com/elastic/beats/blob/main/go.mod#L419C96-L420C2

[mauri870]

I don't think relying on a fork from an employee's personal GitHub account is the right approach. I'd prefer to have these under Elastic's organization, as it would be easier to manage and maintain in the long run. According to elastic/beats#41084, we don't want to keep temporary replacements around for too long. We should make our best effort to upstream any fixes or switch to/maintain another up-to-date fork.

[blakerouse]

100% true. But blocking this PR for something that is already true for beats, doesn't provide forward progress. Lets push for progress and make adjustments a long the way.

[leehinman]

The sarama one was already there. Added fsnotify, we won't need it until we bring in Auditbeat, but be might as well start the pain now so we have more motivation to get those changes pushed upstream.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 3489_beat_receiver2 upstream/3489_beat_receiver2
git merge upstream/main
git push upstream 3489_beat_receiver2

[leehinman]

Seems we where doing double work on this one #5666

You got the integration test added before me, so I am going to close mine in favor of yours.

Sorry, didn't see that you had started. The integration test took more time than the code changes :-)

[blakerouse]

@leehinman No big deal. You will need to add this change to your PR as well - https://github.com/elastic/elastic-agent/pull/5666/files#diff-d002edd1b34ea944ff5d1278c97e55f0b42a7240299fa6e349843687b4d1144cR21

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[blakerouse]

🎉