Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Elasticsearch module on Kubernetes #15707

Closed
PhaedrusTheGreek opened this issue Jan 21, 2020 · 1 comment
Closed

Support for Elasticsearch module on Kubernetes #15707

PhaedrusTheGreek opened this issue Jan 21, 2020 · 1 comment
Labels
Filebeat Filebeat Metricbeat Metricbeat Stalled Team:Integrations Label for the Integrations team

Comments

@PhaedrusTheGreek
Copy link
Contributor

PhaedrusTheGreek commented Jan 21, 2020

I've tried a few approaches to try to use Filebeat and Metricbeat's elasticsearch module in an ECK environment to no avail. Here are the issues as I have found them:

Metricbeat

  • Metricbeat's elasticsearch module must need to run on a 1:1 basis for each Elasticsearch pod, so perhaps a side-car deployment without any changes to the module would work.

Filebeat

  • The default kubernetes logs ingestion doesn't seem to work for a number of reasons, which I assume are mostly due to the elasticsearch module being required as the authority on format/enrichment for integrations such as monitoring.

  • Autodiscover with module support didn't seem to work but I'm unclear on the issues why. It seems that Kubernetes autodiscovery is a higher level problem, and that the current issue is a matter of making the module compatible with Kubernetes.

  • The elasticsearch module is supposed to auto-configure to it's environment, but mistakenly picks up Redhat as the environment , and looks for /var/log/elasticsearch/*.

  • In Kubernetes, all logs are in a single file differentiated by the type key. Normal Elasticsearch logs in JSON format also follow this pattern, but are separated into different files. The module doesn't seem to support a single-file input.

There is some related information here but not specific to the monitoring & logging modules.

@PhaedrusTheGreek PhaedrusTheGreek added Filebeat Filebeat Metricbeat Metricbeat Team:Integrations Label for the Integrations team labels Jan 21, 2020
blakerouse added a commit to blakerouse/beats that referenced this issue Jan 27, 2020
…tdout is selected. (elastic#15707)

* Disable default logger when stdlog or the default all selector is not enabled.

* Fix issue where default go logger is not discarded when either * or stdout is selected. Fix elastic#10251.

* Change logic to always discard unless in debug and when all or stdlog is selected. Add more tests.

* Fix comments.

* Add changelog entry.

(cherry picked from commit 1838734)
blakerouse added a commit that referenced this issue Jan 28, 2020

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
…tdout is selected. (#15707) (#15880)

* Disable default logger when stdlog or the default all selector is not enabled.

* Fix issue where default go logger is not discarded when either * or stdout is selected. Fix #10251.

* Change logic to always discard unless in debug and when all or stdlog is selected. Add more tests.

* Fix comments.

* Add changelog entry.

(cherry picked from commit 1838734)

Co-authored-by: kaiyan-sheng <[email protected]>
@botelastic
Copy link

botelastic bot commented Dec 27, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@botelastic botelastic bot added the Stalled label Dec 27, 2020
@botelastic botelastic bot closed this as completed Jan 26, 2021
@zube zube bot removed the [zube]: Done label Apr 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Filebeat Filebeat Metricbeat Metricbeat Stalled Team:Integrations Label for the Integrations team
Projects
None yet
Development

No branches or pull requests

2 participants