Uplifting Kubernetes Standalone Manifest #2113

gizas · 2023-01-17T09:28:13Z

Cloudantive team hosts and is responsible to housekeep the standalone version of https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml.
As part of this continuous effort we uplifted the Stanadlone Manifest in order to include latest changes for System and Kubernetes Integration

What does this PR do?

Uplifting versions of System Integration and Kubernetes Integration into our manifest. Releavant changes to conditions and missing datastreams added

Why is it important?

In order to keep standalone manifest consistent with latest changes of our integration pacakges.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool

How to test this PR locally

…nto our manifest

mergify · 2023-01-17T09:29:22Z

This pull request does not have a backport label. Could you fix it @gizas? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2023-01-17T09:47:49Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-01-17T09:28:29.886+0000
Duration: 19 min 14 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4825
Skipped	13
Total	4838

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages.
run integration tests : Run the Elastic Agent Integration tests.
run end-to-end tests : Generate the packages and run the E2E Tests.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-01-17T09:47:59Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	98.305% (`58/59`)	👍
Files	69.268% (`142/205`)	👍
Classes	69.231% (`270/390`)	👍
Methods	54.047% (`828/1532`)	👍
Lines	39.246% (`9005/22945`)	👍 0.031
Conditionals	100.0% (`0/0`)	💚

gsantoro · 2023-01-17T10:19:57Z

Some errors are generated when running the current standalone manifest

{
    "log.level": "error",
    "@timestamp": "2023-01-17T10:00:07.412Z",
    "log.origin": {
        "file.name": "coordinator/coordinator.go",
        "file.line": 833
    },
    "message": "Unit state changed beat/metrics-monitoring-metrics-monitoring-beats (CONFIGURING->FAILED): [output unit has no config]",
    "component": {
        "id": "beat/metrics-monitoring",
        "state": "HEALTHY"
    },
    "unit": {
        "id": "beat/metrics-monitoring-metrics-monitoring-beats",
        "type": "input",
        "state": "FAILED",
        "old_state": "CONFIGURING"
    },
    "ecs.version": "1.6.0"
}

There is already an issue open about the previous error at #2086

gsantoro · 2023-01-17T10:21:53Z

Another error

{
    "log.level": "error",
    "@timestamp": "2023-01-17T09:59:53.950Z",
    "message": "Error fetching data for metricset beat.stats: monitored beat is using Elasticsearch output but cluster UUID cannot be determined",
    "component": {
        "binary": "metricbeat",
        "dataset": "elastic_agent.metricbeat",
        "id": "beat/metrics-monitoring",
        "type": "beat/metrics"
    },
    "ecs.version": "1.6.0",
    "log.origin": {
        "file.line": 256,
        "file.name": "module/wrapper.go"
    },
    "service.name": "metricbeat",
    "ecs.version": "1.6.0"
}

Issue at elastic/beats#34217

gsantoro · 2023-01-17T10:24:30Z

and a warning

{
    "log.level": "warn",
    "@timestamp": "2023-01-17T09:59:53.931Z",
    "message": "error when check request status for getting IMDSv2 token: http request status 405. No token in the metadata request will be used.",
    "component": {
        "binary": "metricbeat",
        "dataset": "elastic_agent.metricbeat",
        "id": "beat/metrics-monitoring",
        "type": "beat/metrics"
    },
    "log.logger": "add_cloud_metadata",
    "log.origin": {
        "file.line": 97,
        "file.name": "add_cloud_metadata/provider_aws_ec2.go"
    },
    "service.name": "metricbeat",
    "ecs.version": "1.6.0",
    "ecs.version": "1.6.0"
}

Issue at elastic/beats#33058

MichaelKatsoulis · 2023-01-17T10:28:27Z

deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml

@@ -35,7 +35,7 @@ data:
        meta:
          package:
            name: kubernetes
-            version: 1.9.0
+            version: 1.29.2


Are we supposed to update this every time we have a new version of the integration?

it would be great to know how you generated this manifest for next time

For now I went to UI and created the standalone version of manifest from there. This included latest references for packages and manually I did the changes.

Are we supposed to update this every time we have a new version of the integration?

Good one, I don think we can follow the system integration changes all the time. I think we should try to keep updating yes but still trying to figure out how can we constantly. Maybe periodically once per quarter? Or from now on if an important k8s integration is added to have it in mind?

I think that the standalone configuration only makes sense to follow the integration's changes in case the data_streams' inputs are changed. For example a new version of the integration that just changes some visualisations or the package wording has no impact to the standalone input. So maybe a best effort approach here would be fine, and integration maintainers/contributors should have in mind to update this manifest whenever the integration/package has an input-level change.

MichaelKatsoulis · 2023-01-17T10:29:53Z

deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml

-                    ecs.version: 1.12.0
+              - add_locale: null
+            ignore_older: 72h
+      - id: windows-event-log


Any background on this change?

@elastic/elastic-agent-data-plane can you give us some more info about this change?

We updated our standalone manifests to include your last package version and we are in the progress of aligning the manifest that is auto-generated from kibana UI with what we have in github. So we removed the add_field processor but we added the add_locale that we found. Can you give us some more info? Is it needed?

The `condition: '${host.platform} == ''windows''' will evaluate to false for every stream and prevent this from actually starting.

~~For Kubernetes you can omit this since Windows isn't supported. The policy generated by Fleet is universal for all operating systems.~~

Edit: Windows containers are are a thing now so this can stay: https://kubernetes.io/docs/concepts/windows/intro/

You want to align this as closely as you can with the latest version of the system package generated by Fleet, as that is what is actually maintained.

I think there is a problem here where this is naturally going to drift out of sync with the latest version of the system package unless we either automate updates to it or do the manual work to sync it.

What do you actually want to have in these manifests? What is the goal? It seems like there would be less chance of regression if we only included a minimal configuration that is kept up to date.

Is it worth including only the Kubernetes integration here and omitting the system integration entirely?

@cmacknz Sorry for not replying you before, Indeed on one hand we have the problem of syncing and on the other one we want to have it minimal especially no automation is implemented at the moment.

For the moment we agreed to keep the best effort and manually do the sync and also to keep system integration for now.

| Is it worth including only the Kubernetes integration here and omitting the system integration entirely?

This is something that might reconsider

gsantoro · 2023-01-17T10:34:18Z

FYI I get the same warning "error when check request status for getting IMDSv2 token" when elastic-agent is running in managed mode. I think it has to do with the fact that I am running on GCP and that error is complaining about AWS.

So maybe we can ignore this warning since it is not specific to the standalone mode.

gizas · 2023-01-18T08:31:52Z

deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml

            metricsets:
              - process
            process.include_cpu_ticks: false
-            system.hostfs: /hostfs


@elastic/elastic-agent-data-plane we also removed this. Is not needed in latest 1.20.4 version right?

There were a collection of changes that made system.hostfs: /hostfs unnecessary. @fearful-symmetry can confirm.

gizas · 2023-01-18T08:32:51Z

deploy/kubernetes/elastic-agent-standalone/elastic-agent-standalone-daemonset-configmap.yaml

@@ -433,6 +447,7 @@ data:
          - data_stream:
              dataset: system.load
              type: metrics
+            condition: '${host.platform} != ''windows'''


@elastic/elastic-agent-data-plane is this also correct? Why we exclude windows in system load?

Likely the underlying Metricbeat doesn't implement collection of it.

gizas · 2023-01-18T08:35:15Z

There is already an issue open about the previous error at #2086
@gsantoro for this error, was it because you reapplied the manifest? So in clean environment do you still see it?

Trying to reproduce it

gizas · 2023-01-18T08:37:16Z

Issue at elastic/beats#34217

We can also ignore it. It dissapears after connection

ChrsMark

Changes lgtm!

I'm not familiar with the system integration details though but since we use the latest version of the package itselft, we could consider it as valid?

ChrsMark · 2023-01-22T23:59:38Z

deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml

@@ -35,7 +35,7 @@ data:
        meta:
          package:
            name: kubernetes
-            version: 1.9.0
+            version: 1.29.2


I think that the standalone configuration only makes sense to follow the integration's changes in case the data_streams' inputs are changed. For example a new version of the integration that just changes some visualisations or the package wording has no impact to the standalone input. So maybe a best effort approach here would be fine, and integration maintainers/contributors should have in mind to update this manifest whenever the integration/package has an input-level change.

Uplifting versions of System Integration and Kubernetes Integration i…

8ca00e1

…nto our manifest

gizas requested a review from a team as a code owner January 17, 2023 09:28

gizas requested review from gsantoro and MichaelKatsoulis and removed request for a team January 17, 2023 09:28

mergify bot assigned gizas Jan 17, 2023

mergify bot added the backport-skip label Jan 17, 2023

MichaelKatsoulis reviewed Jan 17, 2023

View reviewed changes

gsantoro added a commit to gsantoro/elastic-agent that referenced this pull request Jan 17, 2023

updated config map for standalone from elastic#2113

895dc68

gizas commented Jan 18, 2023

View reviewed changes

ChrsMark approved these changes Jan 23, 2023

View reviewed changes

gizas merged commit 7ebe691 into main Jan 24, 2023

gizas deleted the standalonemanifests branch January 24, 2023 15:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uplifting Kubernetes Standalone Manifest #2113

Uplifting Kubernetes Standalone Manifest #2113

gizas commented Jan 17, 2023

mergify bot commented Jan 17, 2023

elasticmachine commented Jan 17, 2023

Build stats

Test stats 🧪

elasticmachine commented Jan 17, 2023

gsantoro commented Jan 17, 2023

gsantoro commented Jan 17, 2023

gsantoro commented Jan 17, 2023

MichaelKatsoulis Jan 17, 2023

gsantoro Jan 17, 2023

gizas Jan 18, 2023

gizas Jan 18, 2023

ChrsMark Jan 22, 2023

MichaelKatsoulis Jan 17, 2023

gizas Jan 18, 2023

cmacknz Jan 19, 2023 •

edited

Loading

cmacknz Jan 19, 2023

gizas Jan 26, 2023

gsantoro commented Jan 17, 2023

gizas Jan 18, 2023

cmacknz Jan 19, 2023

gizas Jan 18, 2023

cmacknz Jan 19, 2023

gizas commented Jan 18, 2023

gizas commented Jan 18, 2023

ChrsMark left a comment

ChrsMark Jan 22, 2023

Uplifting Kubernetes Standalone Manifest #2113

Uplifting Kubernetes Standalone Manifest #2113

Conversation

gizas commented Jan 17, 2023

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

mergify bot commented Jan 17, 2023

elasticmachine commented Jan 17, 2023

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Jan 17, 2023

🌐 Coverage report

gsantoro commented Jan 17, 2023

gsantoro commented Jan 17, 2023

gsantoro commented Jan 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmacknz Jan 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gsantoro commented Jan 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gizas commented Jan 18, 2023

gizas commented Jan 18, 2023

ChrsMark left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmacknz Jan 19, 2023 •

edited

Loading