Central Management #7028

exekias · 2018-05-07T10:07:42Z

Beats central management will allow users to configure & manage Beats centrally from Kibana. This is a meta issue to group all efforts for the first version of this feature:

Fixes:

Bug when unenrolling a beats Can't unenroll beats #9452

Tasks:

The text was updated successfully, but these errors were encountered:

exekias · 2018-05-07T17:20:57Z

Pinging @ycombinator @brandonmensing for awareness

This PR unifies all uses of dynamic runner handling under a new `RunnerList` structure. It can receive a list of configs as the desired state and do all the needed changes (start/stopping) to transition to it. I plan to use this as part of #7028

This PR implements intial enrollment to Central Management in Kibana. After running the enrollment command, beats will have a valid access token to use when retrieving configurations. To test this: - Use the following branches: - Elasticsearch: https://github.com/ycombinator/elasticsearch/tree/x-pack/management/beats - Kibana: https://github.com/elastic/kibana/tree/feature/x-pack/management/beats - Retrieve a valid enrollment token: ``` curl \ -u elastic \ -H 'kbn-xsrf: foobar' \ -H 'Content-Type: application/json' \ -X POST \ http://localhost:5601/api/beats/enrollment_tokens ``` - Use it: ``` <beat> enroll http://localhost:5601 <enrollment_token> ``` - Check agent is enrolled: ``` curl http://localhost:5601/api/beats/agents | jq ``` This is part of #7028, closes #7032

exekias · 2018-10-03T22:35:52Z

Pull request to merge Central Management branch to Beats master is now open: #8559

I plan to backport elastic/elasticsearch#30520 & elastic/stack-docs#80 to 6.x tomorrow, I may need a hand with that if you have some time @ycombinator 😺

* Beats enrollment subcommand (#7182) This PR implements intial enrollment to Central Management in Kibana. After running the enrollment command, beats will have a valid access token to use when retrieving configurations. To test this: - Use the following branches: - Elasticsearch: https://github.com/ycombinator/elasticsearch/tree/x-pack/management/beats - Kibana: https://github.com/elastic/kibana/tree/feature/x-pack/management/beats - Retrieve a valid enrollment token: ``` curl \ -u elastic \ -H 'kbn-xsrf: foobar' \ -H 'Content-Type: application/json' \ -X POST \ http://localhost:5601/api/beats/enrollment_tokens ``` - Use it: ``` <beat> enroll http://localhost:5601 <enrollment_token> ``` - Check agent is enrolled: ``` curl http://localhost:5601/api/beats/agents | jq ``` This is part of #7028, closes #7032 * Add API client to retrieve configurations from CM (#8155) * Add central management service (#8263) * Add config manager initial skeleton Config manager will poll configs from Kibana and apply them locally. It must be started with the beat. In order to check the user is not trying to override configurations provided by central management, the Config Manager can check the exisitng configuration and return errors if something is wrong. * Register output for reloading (#8378) * Also send beat name when enrolling (#8380) * Refactor how configs are stored (#8379) * Refactor configs storage to avoid YAML issues * Refactor manager loop to avoid repeated code * Use beat name var when registering confs (#8435) This should make Auditbeat or any other beat based on Metricbeat have their own namespace for confs * Allow user/passwd based enrollment (#8524) * Allow user/passwd based enrollment This allows to enroll using the following workflow: ``` $ <beat> enroll http://kibana:5601 --username elastic Enter password: Enrolled and ready to retrieve settings from Kibana ``` It also allows to pass the password as an env variable: ``` PASS=... $ <beat> enroll http://kibana:5601 --username elastic --password env:PASS Enrolled and ready to retrieve settings from Kibana ``` * Fix some strings after review comments * Add changelog

ycombinator · 2018-10-04T14:07:10Z

@exekias Backport PRs are up: elastic/elasticsearch#34305 and elastic/stack-docs#124.

* Beats enrollment subcommand (elastic#7182) This PR implements intial enrollment to Central Management in Kibana. After running the enrollment command, beats will have a valid access token to use when retrieving configurations. To test this: - Use the following branches: - Elasticsearch: https://github.com/ycombinator/elasticsearch/tree/x-pack/management/beats - Kibana: https://github.com/elastic/kibana/tree/feature/x-pack/management/beats - Retrieve a valid enrollment token: ``` curl \ -u elastic \ -H 'kbn-xsrf: foobar' \ -H 'Content-Type: application/json' \ -X POST \ http://localhost:5601/api/beats/enrollment_tokens ``` - Use it: ``` <beat> enroll http://localhost:5601 <enrollment_token> ``` - Check agent is enrolled: ``` curl http://localhost:5601/api/beats/agents | jq ``` This is part of elastic#7028, closes elastic#7032 * Add API client to retrieve configurations from CM (elastic#8155) * Add central management service (elastic#8263) * Add config manager initial skeleton Config manager will poll configs from Kibana and apply them locally. It must be started with the beat. In order to check the user is not trying to override configurations provided by central management, the Config Manager can check the exisitng configuration and return errors if something is wrong. * Register output for reloading (elastic#8378) * Also send beat name when enrolling (elastic#8380) * Refactor how configs are stored (elastic#8379) * Refactor configs storage to avoid YAML issues * Refactor manager loop to avoid repeated code * Use beat name var when registering confs (elastic#8435) This should make Auditbeat or any other beat based on Metricbeat have their own namespace for confs * Allow user/passwd based enrollment (elastic#8524) * Allow user/passwd based enrollment This allows to enroll using the following workflow: ``` $ <beat> enroll http://kibana:5601 --username elastic Enter password: Enrolled and ready to retrieve settings from Kibana ``` It also allows to pass the password as an env variable: ``` PASS=... $ <beat> enroll http://kibana:5601 --username elastic --password env:PASS Enrolled and ready to retrieve settings from Kibana ``` * Fix some strings after review comments * Add changelog (cherry picked from commit 4247bc3)

* Add Central Management feature (#8559) * Beats enrollment subcommand (#7182) This PR implements intial enrollment to Central Management in Kibana. After running the enrollment command, beats will have a valid access token to use when retrieving configurations. To test this: - Use the following branches: - Elasticsearch: https://github.com/ycombinator/elasticsearch/tree/x-pack/management/beats - Kibana: https://github.com/elastic/kibana/tree/feature/x-pack/management/beats - Retrieve a valid enrollment token: ``` curl \ -u elastic \ -H 'kbn-xsrf: foobar' \ -H 'Content-Type: application/json' \ -X POST \ http://localhost:5601/api/beats/enrollment_tokens ``` - Use it: ``` <beat> enroll http://localhost:5601 <enrollment_token> ``` - Check agent is enrolled: ``` curl http://localhost:5601/api/beats/agents | jq ``` This is part of #7028, closes #7032 * Add API client to retrieve configurations from CM (#8155) * Add central management service (#8263) * Add config manager initial skeleton Config manager will poll configs from Kibana and apply them locally. It must be started with the beat. In order to check the user is not trying to override configurations provided by central management, the Config Manager can check the exisitng configuration and return errors if something is wrong. * Register output for reloading (#8378) * Also send beat name when enrolling (#8380) * Refactor how configs are stored (#8379) * Refactor configs storage to avoid YAML issues * Refactor manager loop to avoid repeated code * Use beat name var when registering confs (#8435) This should make Auditbeat or any other beat based on Metricbeat have their own namespace for confs * Allow user/passwd based enrollment (#8524) * Allow user/passwd based enrollment This allows to enroll using the following workflow: ``` $ <beat> enroll http://kibana:5601 --username elastic Enter password: Enrolled and ready to retrieve settings from Kibana ``` It also allows to pass the password as an env variable: ``` PASS=... $ <beat> enroll http://kibana:5601 --username elastic --password env:PASS Enrolled and ready to retrieve settings from Kibana ``` * Fix some strings after review comments * Add changelog (cherry picked from commit 4247bc3) * Fix monitoring registry usage

ph · 2019-01-23T21:43:38Z

I will close this so we can track the remaining individual items instead.

This PR implements intial enrollment to Central Management in Kibana. After running the enrollment command, beats will have a valid access token to use when retrieving configurations. To test this: - Use the following branches: - Elasticsearch: https://github.com/ycombinator/elasticsearch/tree/x-pack/management/beats - Kibana: https://github.com/elastic/kibana/tree/feature/x-pack/management/beats - Retrieve a valid enrollment token: ``` curl \ -u elastic \ -H 'kbn-xsrf: foobar' \ -H 'Content-Type: application/json' \ -X POST \ http://localhost:5601/api/beats/enrollment_tokens ``` - Use it: ``` <beat> enroll http://localhost:5601 <enrollment_token> ``` - Check agent is enrolled: ``` curl http://localhost:5601/api/beats/agents | jq ``` This is part of elastic#7028, closes elastic#7032

exekias added the meta label May 7, 2018

ruflin assigned exekias May 7, 2018

exekias assigned ycombinator and unassigned ycombinator May 7, 2018

ycombinator mentioned this issue May 9, 2018

Beats central management: index templates and built-in roles elastic/elasticsearch#30493

Closed

This was referenced May 24, 2018

Unify Runner dynamic launching and reloading under RunnerList #7172

Merged

Beats enrollment subcommand #7182

Merged

This was referenced Jul 10, 2018

Documents built-in beats_admin role elastic/stack-docs#80

Merged

[X-Pack] Beats centralized management: security role + licensing elastic/elasticsearch#30520

Merged

exekias added the enhancement label Jul 19, 2018

mattapperson assigned mattapperson and justinkambic Aug 6, 2018

exekias mentioned this issue Sep 14, 2018

Add central management service #8263

Merged

andrewkroh mentioned this issue Sep 18, 2018

cluster filebeat #5212

Closed

dedemorton mentioned this issue Sep 27, 2018

Add docs and edit strings for Beats Central Management #8470

Closed

9 tasks

alvarolobato mentioned this issue Nov 12, 2018

APM Server should support Beats central configuration elastic/apm-server#1539

Closed

ph self-assigned this Dec 11, 2018

ph added the Management label Dec 12, 2018

ph closed this as completed Jan 23, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Central Management #7028

Central Management #7028

exekias commented May 7, 2018 •

edited by ph

Loading

Tasks currently awaiting efforts from other teams

Overlay work (tasks that do not impede a working demo/beta)

exekias commented May 7, 2018

exekias commented Oct 3, 2018

ycombinator commented Oct 4, 2018

ph commented Jan 23, 2019

Central Management #7028

Central Management #7028

Comments

exekias commented May 7, 2018 • edited by ph Loading

Tasks currently awaiting efforts from other teams

Overlay work (tasks that do not impede a working demo/beta)

exekias commented May 7, 2018

exekias commented Oct 3, 2018

ycombinator commented Oct 4, 2018

ph commented Jan 23, 2019

exekias commented May 7, 2018 •

edited by ph

Loading