Add Elastic Agent #1005

graphaelli · 2020-12-11T03:17:26Z

What does this PR do?

Adds Elastic Agent into apm-integration-testing using --with-elastic-agent option. Elastic Agent is configured in fleet mode with system logs and metrics integrations, the usual defaults.

Why is it important?

For testing integrations with Elastic Agent eg APM.

Some things to note:

elastic-agent.yml can't be bind mounted into the container as agent tries to move it out of the way when fleeting. As a result it seems any customizations will have to happen after the agent is running - this will be follow as a separate effort for Add Elastic Agent container #1004.
The kibana instance in apm-it will direct agents to use it for fleet along with the configured elasticsearch - support for using a different kibana seems unimportant right now
The setup and enroll commands aren't silent so docker logs for this container are not nice. This message seems to just be a warning: Elastic Agent might not be running; unable to trigger restart
Elastic agent logs to various files within the container and not to stdout like usual docker logging - it ships its own logs so that's nice but it's surprising while debugging unless you know this is happening
The APM integration is listed but can't be installed, currently showing the following. This does not block this pr:

Related issues

related #1004

This reverts commit b3e5723.

assuming it can resolve hostnames aka is also running under docker in the same network

apmmachine · 2020-12-11T03:25:05Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Started by timer
Start Time: 2020-12-14T04:54:00.768+0000
Duration: 35 min 49 sec

Test stats 🧪

Test	Results
Failed	0
Passed	442
Skipped	12
Total	454

simitt · 2020-12-11T10:08:48Z

Great progress!

The APM integration is listed but can't be installed, currently showing the following. This does not block this pr:

I managed to install the APM integration via navigating to Policies/Add integration.

Something is odd when downloading the packages: on the first attempt it always returns a 404 for the 0.1.0-dev1 package, on second try it finds the APM package and downloads it successfully. The package was added by @jalvz yesterday, I don't see why it shouldn't be found on first try.

The APM Server is not yet bundled nor available via the artifacts API. While I managed to install the APM integration, the APM Server cannot actually start as it is not yet available.

simitt · 2020-12-11T10:39:18Z

The agent could successfully enroll, also be assigned to a new policy; but when installing the APM integration to the policy, the agent went offline soon, as it couldn't successfully checkin with fleet; resulting in following log error:

{"log.level":"error","@timestamp":"2020-12-11T10:10:45.187Z","log.origin":{"file.name":"application/fleet_gateway.go","file.line":176},"message":"Could not communicate with Checking API will retry, error: fail to checkin to fleet: Post \"http://0.0.0.0:5601/api/fleet/agents/d216e510-3b8f-11eb-b13c-0ddb183c8890/checkin?\": dial tcp 0.0.0.0:5601: connect: connection refused","ecs.version":"1.6.0"}

Not an issue with this PR, just leaving the comment here in case others try it out too - needs to be fixed separately.

simitt · 2020-12-11T10:44:21Z

In another attempt, starting with completely fresh docker images - the APM Server integration was now successfully installed, without any further changes.

(0) Starting integration testing with your branch and elastic-agent enabled
(1) Creating a new policy and assigning policy to agent
(2) Adding APM to the policy
Error on first attempt to check APM integration, and on first attempt to apply changes:

Succesful on second attempt (similar as mentioned in previous comment):

APM Server is now running, and can be reached from inside the elastic agent docker container (port is not opened yet)

graphaelli · 2020-12-13T02:56:04Z

Nice @simitt! I haven't been able to get it to work adding the integration via a different flow:

I will try the method you suggested. Any changes need to this pull request?

graphaelli · 2020-12-13T03:02:39Z

I was able to reproduce your result by installing the integration from the policy page:

simitt

Great to get this in! Just left minor comments - feel free to skip.

simitt · 2020-12-14T07:42:53Z

scripts/modules/elastic_stack.py

+        # FLEET_TOKEN_NAME - token name for a token to be created
+        # KIBANA_HOST - actual kibana host [http://localhost:5601]
+        # KIBANA_PASSWORD - password for accessing kibana API [changeme]
+        # KIBANA_USERNAME - username for accessing kibana API [elastic]


isn't that admin instead of elastic?

simitt · 2020-12-14T07:43:36Z

scripts/modules/elastic_stack.py

+        parser.add_argument(
+            "--elastic-agent-kibana-url",
+            default="http://admin:changeme@" + cls.DEFAULT_KIBANA_HOST,
+            help="Elastic Agent's Kibana URL"


maybe add including username + password

jalvz · 2020-12-14T16:23:30Z

Thanks @graphaelli and @simitt !
I could reproduce the same outcome as you....

jalvz · 2020-12-14T16:53:40Z

scripts/modules/elastic_stack.py

@@ -859,6 +922,10 @@ def __init__(self, **options):
            if self.at_least_version("7.7"):
                self.environment["XPACK_SECURITY_ENCRYPTIONKEY"] = "fhjskloppd678ehkdfdlliverpoolfcr"
                self.environment["XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY"] = "fhjskloppd678ehkdfdlliverpoolfcr"
+            if self.at_least_version("7.8"):
+                self.environment["XPACK_FLEET_AGENTS_ELASTICSEARCH_HOST"] = urls[0]


The Elastic Agent container crashes in 7.8, and I suspect it might be because these settings were named differently. I just can't find it, https://www.elastic.co/guide/en/kibana/7.10/fleet-settings-kb.html doesn't exist for previous versions...

We might just as well support >=7.10 only and revisit if needed for earlier version?

The error is pretty clear, thou:

localtesting_7.8.1_elastic-agent | {"statusCode":404,"error":"Not Found","message":"Not Found"}{"statusCode":404,"error":"Not Found","message":"Not Found"}{"statusCode":404,"error":"Not Found","message":"Not Found"}

Sounds good to me, perhaps @simitt can include it as part of the next chunk of #1004

graphaelli added 10 commits December 9, 2020 16:58

bare minimum

1e6d775

stub healthcheck

b9bba2f

enroll elastic agent

c6f0967

restrict agent version

355a3ed

use elastic-agent.yml

b3e5723

fix kibana url option

1ee384c

Revert "use elastic-agent.yml"

b9a659e

This reverts commit b3e5723.

provide docker.sock

48a1607

help agent find kibana and es

3ef764a

assuming it can resolve hostnames aka is also running under docker in the same network

more agent tests

0c77461

fix test

d067c77

simitt approved these changes Dec 14, 2020

View reviewed changes

simitt mentioned this pull request Dec 14, 2020

[Fleet][APM] APM Server via Elastic Agent Fleet is missing permissions to create data stream elastic/kibana#85761

Closed

clarify text per review feedback

356ce54

graphaelli merged commit 82efeb7 into elastic:master Dec 14, 2020

graphaelli deleted the elastic-agent branch December 14, 2020 14:33

graphaelli mentioned this pull request Dec 14, 2020

Add Elastic Agent container #1004

Closed

jalvz reviewed Dec 14, 2020

View reviewed changes

This was referenced Dec 16, 2020

Integrate with Elastic Agent elastic/apm-server#4004

Closed

Add package-registry container #1010

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Elastic Agent #1005

Add Elastic Agent #1005

graphaelli commented Dec 11, 2020 •

edited

Loading

apmmachine commented Dec 11, 2020 •

edited

Loading

Build stats

Test stats 🧪

simitt commented Dec 11, 2020

simitt commented Dec 11, 2020

simitt commented Dec 11, 2020

graphaelli commented Dec 13, 2020

graphaelli commented Dec 13, 2020

simitt left a comment

simitt Dec 14, 2020

simitt Dec 14, 2020

jalvz commented Dec 14, 2020 •

edited

Loading

jalvz Dec 14, 2020

graphaelli Dec 15, 2020

Add Elastic Agent #1005

Add Elastic Agent #1005

Conversation

graphaelli commented Dec 11, 2020 • edited Loading

What does this PR do?

Why is it important?

Related issues

apmmachine commented Dec 11, 2020 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

simitt commented Dec 11, 2020

simitt commented Dec 11, 2020

simitt commented Dec 11, 2020

graphaelli commented Dec 13, 2020

graphaelli commented Dec 13, 2020

simitt left a comment

Choose a reason for hiding this comment

simitt Dec 14, 2020

Choose a reason for hiding this comment

simitt Dec 14, 2020

Choose a reason for hiding this comment

jalvz commented Dec 14, 2020 • edited Loading

jalvz Dec 14, 2020

Choose a reason for hiding this comment

graphaelli Dec 15, 2020

Choose a reason for hiding this comment

graphaelli commented Dec 11, 2020 •

edited

Loading

apmmachine commented Dec 11, 2020 •

edited

Loading

jalvz commented Dec 14, 2020 •

edited

Loading