[Agent] Grpc support for agent to beat communication #14835

michalpristas · 2019-11-28T08:58:11Z

New configuration option is introduced specifying the mode of management.
It can either be configured to central management [cm] or [fleet]

if fleet is configured grpc server is started listening to specified address acquired during initial handshake with agent.

Fixes #14449

michalpristas · 2019-11-29T11:44:34Z

To test this you need to:

specify "grpc" in {beat}.spec
enabled managment in {beat}.yml
- management.enabled: true
- management.mode: fleet

Agent will start and forward config, in next step we need to make sure metricbeat and filebeat understands it using correct reloadables. I propose this to be separate PR

libbeat/cmd/instance/config_server.go

ph · 2019-12-05T14:25:29Z

Currently reviewing this.

michalpristas · 2019-12-05T16:31:09Z

Jenkins test this please

ph · 2019-12-05T20:16:01Z

filebeat/filebeat.spec

@@ -0,0 +1 @@
+{"BinaryPath":"filebeat","Args":["-e"],"Configurable":"grpc"}


Can we move that to YAML spec files?

do you mean classic filebeat.yml?

I am talking about the files in x-pack/agent/spec.

x-pack/agent/spec are transformation rules. this file should describe specific binary.
if we stick to agent.version == beat.version it is fine to have it there. but we probably still need to support {program}.spec for not yet known programs. e.g running endpoint or community beat eventually, so agent knows if specific binary supports grpc protocol or if there are some arguments needed to start binary correctly.
i'm not sure what is a correct way of handling this. maybe lookup local spec + fallback to {program}.spec with local having priority?

For the endpoint case, I think we would just have a specific spec for them. Concerning the community beats its a complete other madness and open the door to all sort of problems. We would need a way to add signatures and support the signing of community beats or expose that. :)

For the POC I think we can keep that way but let's create a followup issue to refactor that this sound like a P3 for the priority.

i created a new issue here: #14985

libbeat/cmd/instance/beat.go

libbeat/cmd/instance/config_server.go

x-pack/libbeat/management/fleet/manager.go

libbeat/management/management.go

x-pack/agent/pkg/agent/operation/operator_handlers.go

houndci-bot · 2019-12-06T16:09:00Z

libbeat/management/management.go

@@ -47,32 +49,46 @@ type ConfigManager interface {
 	CheckRawConfig(cfg *common.Config) error
 }

+type PluginFunc func(*common.Config) FactoryFunc


exported type PluginFunc should have comment or be unexported

libbeat/management/management.go

x-pack/libbeat/management/plugin.go

x-pack/libbeat/management/fleet/plugin.go

ph

Just a few minor comments, I will test it once we get the newly generated packages in.
But we can iterate on that.

ph · 2019-12-10T13:45:56Z

x-pack/libbeat/libbeat.yml

+#================================= Migration ==================================
+
+# This allows to enable 6.7 migration aliases
+#migration.6_to_7.enabled: true


@michalpristas Why this file was generated, there are currently no libbeat.yml in the x-pack/libbeat or the libbeat/ folder?

this somehow sneaked in, woops

ph · 2019-12-10T13:46:14Z

x-pack/libbeat/management/config.go

+	ModeCentralManagement = "x-pack-cm"
+
+	// ModeFleet is a management mode where fleet is used to retrieve configurations
+	ModeFleet = "x-pack-fleet"


Nit: add a period after the sentence.

ph · 2019-12-10T13:47:08Z

x-pack/libbeat/management/fleet/config_server.go

+	configChan chan<- map[string]interface{}
+}
+
+// NewConfigServer creates a new grpc configuration server for receiveing


Suggested change

// NewConfigServer creates a new grpc configuration server for receiveing

// NewConfigServer creates a new grpc configuration server for receiving

ph · 2019-12-10T13:47:39Z

x-pack/libbeat/management/fleet/config_server.go

+}
+
+// NewConfigServer creates a new grpc configuration server for receiveing
+// configurations from Elastic Agent


Suggested change

// configurations from Elastic Agent

// configurations from Elastic Agent.

ph · 2019-12-10T13:47:58Z

x-pack/libbeat/management/fleet/config_server.go

+	}
+}
+
+// Config is a handler of a call made by agent pushing latest configuration


Suggested change

// Config is a handler of a call made by agent pushing latest configuration

// Config is a handler of a call made by agent pushing latest configuration.

ph · 2019-12-10T13:49:48Z

x-pack/libbeat/management/fleet/manager.go

+
+	go m.startGrpcServer()
+
+	return m, nil


lets make a followup issue to merge your rejectlist changes here.

yep i was wondering why it is still named like that

michalpristas added 17 commits November 28, 2019 08:42

initial

5a8be14

yaml parsing

d0612be

registering fleet

67b7c5d

cleanup

f555f26

splitting []iface into []map[string]iface

99e776b

moved from filebeat to libbeat

5ed6102

added vendor

03345f1

removed binaries

c3814f2

missing comments

681b637

metricbeat spec

a63f94f

x-pack/metricbeat supports fleet management

f3e0fb2

global support in x-pack beats for management/fleet

c99a358

added submodule

2578888

start only on run, prevents running on setup or similar commands

ff0b8cd

removed git submodule

2df6934

fixed fleet imports

ee20f38

removed vendor fleet

69ada7e

michalpristas added agent labels Nov 28, 2019

michalpristas added 4 commits November 28, 2019 10:49

fixed tests

0c4d113

fixed tests

990564f

more robust factory method

876f9bf

proper teardown

4bb740f

michalpristas force-pushed the agent branch from dbfd4c3 to 4bb740f Compare November 29, 2019 11:39

michalpristas marked this pull request as ready for review November 29, 2019 11:44

resovled conflicts with fleet

7524167

houndci-bot reviewed Dec 2, 2019

View reviewed changes

libbeat/cmd/instance/config_server.go Outdated Show resolved Hide resolved

houndci-bot reviewed Dec 2, 2019

View reviewed changes

libbeat/cmd/instance/config_server.go Outdated Show resolved Hide resolved

houndci-bot reviewed Dec 2, 2019

View reviewed changes

libbeat/cmd/instance/config_server.go Outdated Show resolved Hide resolved

ph suggested changes Dec 5, 2019

View reviewed changes

michalpristas mentioned this pull request Dec 6, 2019

[Agent] Move beat.spec to baked in spec files #14985

Closed

resolved comments

24ce521

michalpristas changed the title ~~Grpc support for agent to beat communication~~ [Agent] Grpc support for agent to beat communication Dec 6, 2019

michalpristas added 3 commits December 6, 2019 16:08

enabled cm

2b6ab97

set cm

971189f

plugin factory introduced

84e6ddc

houndci-bot reviewed Dec 6, 2019

View reviewed changes

libbeat/management/management.go Show resolved Hide resolved

houndci-bot reviewed Dec 6, 2019

View reviewed changes

libbeat/management/management.go Show resolved Hide resolved

michalpristas added 2 commits December 6, 2019 17:17

i'm nothing but a hound dog

1c1f341

missing files

86c7ba1

houndci-bot reviewed Dec 9, 2019

View reviewed changes

x-pack/libbeat/management/plugin.go Show resolved Hide resolved

x-pack/libbeat/management/fleet/plugin.go Show resolved Hide resolved

houndci-bot reviewed Dec 9, 2019

View reviewed changes

x-pack/libbeat/management/fleet/plugin.go Outdated Show resolved Hide resolved

houndci-bot reviewed Dec 9, 2019

View reviewed changes

x-pack/libbeat/management/fleet/plugin.go Outdated Show resolved Hide resolved

michalpristas force-pushed the agent branch 3 times, most recently from 39a4921 to 655328a Compare December 9, 2019 11:13

comments

06894d2

michalpristas force-pushed the agent branch from 655328a to 06894d2 Compare December 9, 2019 11:16

ph approved these changes Dec 10, 2019

View reviewed changes

resolved comments

d56f46d

ph approved these changes Dec 11, 2019

View reviewed changes

michalpristas added review [zube]: In Review and removed [zube]: In Review labels Dec 11, 2019

michalpristas merged commit 1743bbc into elastic:fleet Dec 11, 2019

leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023

[Agent] Grpc support for agent to beat communication (elastic#14835)

7d1c7ab

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Agent] Grpc support for agent to beat communication #14835

[Agent] Grpc support for agent to beat communication #14835

michalpristas commented Nov 28, 2019 •

edited

Loading

michalpristas commented Nov 29, 2019 •

edited

Loading

ph commented Dec 5, 2019

michalpristas commented Dec 5, 2019

ph Dec 5, 2019

michalpristas Dec 6, 2019

ph Dec 6, 2019

michalpristas Dec 6, 2019

ph Dec 6, 2019

michalpristas Dec 6, 2019

houndci-bot Dec 6, 2019

ph left a comment

ph Dec 10, 2019

michalpristas Dec 11, 2019

ph Dec 10, 2019

ph Dec 10, 2019

ph Dec 10, 2019

ph Dec 10, 2019

ph Dec 10, 2019

michalpristas Dec 11, 2019

		@@ -0,0 +1 @@
		{"BinaryPath":"filebeat","Args":["-e"],"Configurable":"grpc"}

	// NewConfigServer creates a new grpc configuration server for receiveing
	// NewConfigServer creates a new grpc configuration server for receiving

	// configurations from Elastic Agent
	// configurations from Elastic Agent.

	// Config is a handler of a call made by agent pushing latest configuration
	// Config is a handler of a call made by agent pushing latest configuration.

[Agent] Grpc support for agent to beat communication #14835

[Agent] Grpc support for agent to beat communication #14835

Conversation

michalpristas commented Nov 28, 2019 • edited Loading

michalpristas commented Nov 29, 2019 • edited Loading

ph commented Dec 5, 2019

michalpristas commented Dec 5, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ph left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michalpristas commented Nov 28, 2019 •

edited

Loading

michalpristas commented Nov 29, 2019 •

edited

Loading