Elastic Agent not bundling ARM/aarch64 Endpoint #28442

dwhyrock · 2021-10-14T12:48:43Z

The Endpoint team recently added an ARM/aarch64 version of the Endpoint for the 7.16.0 release. It is being picked up in the Release Manager jobs, but it is not currently being bundled with the Elastic Agent.

The Beats/Elastic Agent logic in the Release Manager should be updated to bundle the Endpoint Linux ARM/aarch64 package with Agent.

It's worth noting that the Endpoint file uses aarch64 as the arch string, but some beats use arm64. I'm not sure if some of the logic will need updating in the Release Manager logic for Agent or not.

Related issue: #27197

For confirmed bugs, please report:

Version: 7.16.0
Operating System: Linux
Discuss Forum URL:
Steps to Reproduce:

cc @ferullo

The text was updated successfully, but these errors were encountered:

elasticmachine · 2021-10-14T12:50:40Z

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

elasticmachine · 2021-10-14T13:56:57Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

dwhyrock · 2021-10-19T13:03:06Z

This was sorted out with the above linked issues. Closing.

michalpristas · 2021-10-20T12:49:17Z

with changes in infra in place, endpoint present in arm packages verified

ls elastic-agent-8.0.0-SNAPSHOT-linux-arm64/data/elastic-agent-af602c/downloads 
apm-server-8.0.0-SNAPSHOT-linux-arm64.tar.gz			filebeat-8.0.0-SNAPSHOT-linux-arm64.tar.gz			heartbeat-8.0.0-SNAPSHOT-linux-arm64.tar.gz
apm-server-8.0.0-SNAPSHOT-linux-arm64.tar.gz.asc		filebeat-8.0.0-SNAPSHOT-linux-arm64.tar.gz.asc			heartbeat-8.0.0-SNAPSHOT-linux-arm64.tar.gz.asc
apm-server-8.0.0-SNAPSHOT-linux-arm64.tar.gz.sha512		filebeat-8.0.0-SNAPSHOT-linux-arm64.tar.gz.sha512		heartbeat-8.0.0-SNAPSHOT-linux-arm64.tar.gz.sha512
endpoint-security-8.0.0-SNAPSHOT-linux-arm64.tar.gz		fleet-server-8.0.0-SNAPSHOT-linux-arm64.tar.gz			metricbeat-8.0.0-SNAPSHOT-linux-arm64.tar.gz
endpoint-security-8.0.0-SNAPSHOT-linux-arm64.tar.gz.asc		fleet-server-8.0.0-SNAPSHOT-linux-arm64.tar.gz.asc		metricbeat-8.0.0-SNAPSHOT-linux-arm64.tar.gz.asc
endpoint-security-8.0.0-SNAPSHOT-linux-arm64.tar.gz.sha512	fleet-server-8.0.0-SNAPSHOT-linux-arm64.tar.gz.sha512		metricbeat-8.0.0-SNAPSHOT-linux-arm64.tar.gz.sha512

EricDavisX · 2021-10-22T01:33:40Z

@amolnater-qasource can you take a look to make sure we've validated this from a build candidate please?

amolnater-qasource · 2021-10-25T06:20:50Z

Hi @EricDavisX
We have attempted to install elastic-agent on Linux Ubuntu armx64 (.tar package) and Linux Centos 8.0 armx64 (.rpm package) and we are successfully able to install elastic-agent on 7.16.0 BC-1 Kibana Staging environment.

Build details:
Build: 45504
Commit: 9231d806c9384df4026977ba7435a9302dc2d4ab
Artifact Links:
elastic-agent-7.16.0-linux-arm64.tar.gz
elastic-agent-7.16.0-aarch64.rpm

However when we added Endpoint Security to both the agents we didn't get any endpoint-security logs and no data streams for Endpoint Security.

Further we will revalidate this again on upcoming BC's and please let us know if we are missing anything.

Thanks

ferullo · 2021-10-25T12:48:06Z

Cc @nicholasberlin

EricDavisX · 2021-10-26T01:40:17Z

reopening for tracking.

ferullo · 2021-10-26T01:53:52Z

Hi all, I’m trying to deug why endpoint-security is not starting for arm64.
It seems limited support was added with this PR https://github.com/elastic/beats/pull/28449/files
When I run elastic-agent install --e --v -f … on x86_64 I see
{"log.level":"info","@timestamp":"2021-10-25T18:20:20.973Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-10-25T14:20:20-04:00 - message: Application: endpoint-security--7.16.0[52b7e6ac-c63b-4607-8d34-1b53390ef097]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
However, when I do the same on an arm64 box, I don’t see it.

@nicholasberlin posted that ^ in Slack. I'm not sure a lack of bundling (aka this original issue) is the cause of it because as far as I can tell Endpoint is bundled with the arm64 Agent. But I'm also not sure it isn't the cause. @michalpristas can you help us triage what's going on?

amolnater-qasource · 2021-10-26T05:50:42Z

Hi @EricDavisX
We have revalidated this endpoint installation with Linux arm64 hosts(Ubuntu 20 arm64 and Centos 8 arm64) on 8.0 Snapshot and on 7.16.0 BC-1.

We had below same observations on both the environments:

no logs for endpoint security.
no data under data streams tab.
no Endpoint folder created inside /opt/Elastic

Build details 8.0:
Build: 47507
Commit: 125c5699446346b727a4649f39809457e0076445

Artifact Links:
elastic-agent-8.0.0-SNAPSHOT-linux-arm64.tar.gz
elastic-agent-8.0.0-SNAPSHOT-aarch64.rpm

Please let us know if anything else is required from our end.
Thanks

nicholasberlin · 2021-10-26T11:19:25Z

I strace'd the installation command:
sudo strace ./elastic-agent install --v --e --d debug -f ...
And I see that the endpoint-security tar is being extracted and copied. Things like the following

openat(AT_FDCWD, "/opt/Elastic/Agent/data/elastic-agent-c3a793/downloads/endpoint-security-7.16.0-linux-arm64/NOTICE.txt", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3

$ sudo ls /opt/Elastic/Agent/data/elastic-agent-c3a793/downloads/endpoint-security-7.16.0-linux-arm64
LICENSE.txt  NOTICE.txt  endpoint-security  endpoint-security-resources.zip

nicholasberlin · 2021-10-26T11:43:52Z

I see filebeat and metricbeat but not endpoint-security in the install folder.

root@ip-172-31-43-245:/opt/Elastic/Agent/data/elastic-agent-c3a793# ls install/
filebeat-7.16.0-linux-arm64  metricbeat-7.16.0-linux-arm64

EricDavisX · 2021-10-26T14:00:32Z

I agree the bundling may not be the problem, but want to wait to spawn an issue off until we know what's wrong (and so Where to split it off to). Thanks everyone. @michel-laterman do you have any cycles to poke in to help Michal triage?

michalpristas · 2021-10-27T10:52:28Z

will take a look

amolnater-qasource · 2021-10-29T07:38:06Z

Hi @EricDavisX
Today we don't have a latest snapshot build:

Build details of today's cloud-qa environment:
Build: 45625
Commit: 223397a5ed39a8ab66e369b9a97de4fc7be879e8
Commit details: [October 26, 2021 08:21pm]

Hence, we will be revalidating this whenever we get a new Snapshot build.

Thanks

amolnater-qasource · 2021-11-11T08:50:29Z

Hi @EricDavisX
We have attempted to reinstall endpoint on latest 7.16.0 Snapshot and found it almost working fine.
We installed linux-arm64.tar agent on AWS Ubuntu ARM64 and linux-arm64.rpm agent on AWS Centos8-ARM64.

We added endpoint security integration and both the agents were visible under Endpoints tab.

Endpoint Security- Logs were generated under agent logs tab.
Endpoint Security- data was generated under Data Streams tab.
Error logs were generated for Centos 8 .rpm, for which issue is reported at https://github.com/elastic/endpoint-dev/issues/10135
No issues were observed on Agent or endpoint after reboot.

Build details:
Build: 45910
Commit: af229de1a1dfbe3a65089c75e178351c9d49e68d
Artifacts:
https://snapshots.elastic.co/7.16.0-df026734/downloads/beats/elastic-agent/elastic-agent-7.16.0-SNAPSHOT-aarch64.rpm
https://snapshots.elastic.co/7.16.0-df026734/downloads/beats/elastic-agent/elastic-agent-7.16.0-SNAPSHOT-linux-arm64.tar.gz

Please let us know if anything else is required from our end.
Thanks

EricDavisX · 2021-11-11T21:09:22Z

If the issue noted above is the only thing found then we are in great shape, let's call it done. Thanks!

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 14, 2021

dwhyrock added the Team:Elastic-Agent Label for the Agent team label Oct 14, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 14, 2021

andresrc added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team and removed Team:Elastic-Agent Label for the Agent team labels Oct 14, 2021

jlind23 assigned michalpristas Oct 14, 2021

jlind23 added the 7.16-candidate label Oct 14, 2021

dwhyrock closed this as completed Oct 19, 2021

EricDavisX reopened this Oct 26, 2021

andresrc added the bug label Oct 27, 2021

michalpristas mentioned this issue Oct 27, 2021

[bug][Elastic Agent] Fix endpoint constraint to enable arm64 #28677

Merged

6 tasks

michalpristas closed this as completed in #28677 Oct 28, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Elastic Agent not bundling ARM/aarch64 Endpoint #28442

Elastic Agent not bundling ARM/aarch64 Endpoint #28442

dwhyrock commented Oct 14, 2021

elasticmachine commented Oct 14, 2021

elasticmachine commented Oct 14, 2021

dwhyrock commented Oct 19, 2021

michalpristas commented Oct 20, 2021 •

edited

Loading

EricDavisX commented Oct 22, 2021

amolnater-qasource commented Oct 25, 2021

ferullo commented Oct 25, 2021

EricDavisX commented Oct 26, 2021

ferullo commented Oct 26, 2021

amolnater-qasource commented Oct 26, 2021

nicholasberlin commented Oct 26, 2021

nicholasberlin commented Oct 26, 2021

EricDavisX commented Oct 26, 2021

michalpristas commented Oct 27, 2021

amolnater-qasource commented Oct 29, 2021

amolnater-qasource commented Nov 11, 2021

EricDavisX commented Nov 11, 2021

Elastic Agent not bundling ARM/aarch64 Endpoint #28442

Elastic Agent not bundling ARM/aarch64 Endpoint #28442

Comments

dwhyrock commented Oct 14, 2021

elasticmachine commented Oct 14, 2021

elasticmachine commented Oct 14, 2021

dwhyrock commented Oct 19, 2021

michalpristas commented Oct 20, 2021 • edited Loading

EricDavisX commented Oct 22, 2021

amolnater-qasource commented Oct 25, 2021

ferullo commented Oct 25, 2021

EricDavisX commented Oct 26, 2021

ferullo commented Oct 26, 2021

amolnater-qasource commented Oct 26, 2021

nicholasberlin commented Oct 26, 2021

nicholasberlin commented Oct 26, 2021

EricDavisX commented Oct 26, 2021

michalpristas commented Oct 27, 2021

amolnater-qasource commented Oct 29, 2021

amolnater-qasource commented Nov 11, 2021

EricDavisX commented Nov 11, 2021

michalpristas commented Oct 20, 2021 •

edited

Loading