Allow routing for integrations that are not input packages #6340

gsantoro · 2023-05-26T12:21:09Z

What does this PR do?

This PR add permissions to reroute logs to logs-*-* for integrations that are not input packages.

The full list of packages to edit is:

kubernetes.container_logs
system.syslog
activemq.log
auditd.log
aws.cloudwatch_logs
aws.ec2_logs
kafka.log
docker.container_logs

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Relates Allow routing for integrations that are not input packages #6255

Screenshots

elasticmachine · 2023-05-26T12:36:04Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-06-13T12:11:58.706+0000
Duration: 52 min 27 sec

Test stats 🧪

Test	Results
Failed	0
Passed	531
Skipped	4
Total	535

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-05-26T12:36:13Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`22/22`)	💚
Files	95.833% (`23/24`)	👎 -4.167
Classes	95.833% (`23/24`)	👎 -4.167
Methods	84.645% (`441/521`)	👎 -4.244
Lines	90.563% (`12466/13765`)	👎 -9.437
Conditionals	100.0% (`0/0`)	💚

tommyers-elastic

CSP integration changes look good to me - only thing on my mind is the future - someone looking at this file without the context of routing rules etc, they might be able to look up what these settings do, but there is some implicit coupling to other parts of the system here that i think would be good to call out 'at the source'. can we comment what the settings are for?

packages/activemq/changelog.yml

gsantoro · 2023-05-26T14:00:39Z

about

CSP integration changes look good to me - only thing on my mind is the future - someone looking at this file without the context of routing rules etc, they might be able to look up what these settings do, but there is some implicit coupling to other parts of the system here that i think would be good to call out 'at the source'. can we comment what the settings are for?

I think that we could add a comment like

# Ensures agents have permissions to write data to `logs-nginx.*-*`

like the one you can see at here

gsantoro · 2023-06-13T09:12:34Z

I have tested now the rerouting of logs for the following use cases:

docker.container_logs in 8.9.0-SNAPSHOT Elastic stack
docker.container_logs in 8.8.0 Elastic stack

gsantoro · 2023-06-13T12:11:41Z

/test

elasticmachine · 2023-06-13T12:13:20Z

💚 Build Succeeded

Buildkite Build
Commit: 22eb873

History

💚 Build #21 succeeded 22eb873

cc @gsantoro

elasticmachine · 2023-06-13T13:49:56Z

Package activemq - 0.10.0 containing this change is available at https://epr.elastic.co/search?package=activemq

elasticmachine · 2023-06-13T13:54:34Z

Package auditd - 3.9.0 containing this change is available at https://epr.elastic.co/search?package=auditd

elasticmachine · 2023-06-13T13:58:56Z

Package aws - 1.44.0 containing this change is available at https://epr.elastic.co/search?package=aws

elasticmachine · 2023-06-13T14:03:23Z

Package docker - 2.6.0 containing this change is available at https://epr.elastic.co/search?package=docker

elasticmachine · 2023-06-13T14:08:55Z

Package kafka - 1.7.0 containing this change is available at https://epr.elastic.co/search?package=kafka

elasticmachine · 2023-06-13T14:13:37Z

Package kubernetes - 1.42.0 containing this change is available at https://epr.elastic.co/search?package=kubernetes

elasticmachine · 2023-06-13T14:17:57Z

Package system - 1.33.0 containing this change is available at https://epr.elastic.co/search?package=system

ruflin · 2023-06-15T10:38:29Z

packages/system/manifest.yml

@@ -1,7 +1,7 @@
 format_version: 1.0.0
 name: system
 title: System
-version: 1.32.0-beta.2
+version: 1.33.0


@gsantoro With this change you accidentally release TSDB which was still in beta!

@gsantoro Are we considering System Integration TSDB as GA now or are we planning to revert the change ?

TSDB in system is now approved to be in GA. #6607

* fix system test for kubernetes integration for k8s v1.27.0 * minor changes from formatting * syslong changes * revert a merge conflict * add permissions to selected list of datastreams * update PR id * fix merge conflict with main * Increment minor version instead of bugfix version * added comments for new settings * update activemq version --------- Co-authored-by: Felix Barnsteiner <[email protected]>

gsantoro added 7 commits May 24, 2023 17:10

fix system test for kubernetes integration for k8s v1.27.0

a9d0b0b

Merge branch 'main' into feature/k8s_1.27.0

ac6dbd2

minor changes from formatting

4bb4ecb

Merge remote-tracking branch 'origin' into feature/allow_routing

42fbebb

syslong changes

b545962

Merge remote-tracking branch 'origin' into feature/allow_routing

26a3305

revert a merge conflict

3d1070e

gsantoro added the enhancement New feature or request label May 26, 2023

gsantoro requested a review from a team as a code owner May 26, 2023 12:21

gsantoro self-assigned this May 26, 2023

gsantoro requested review from rdner and cmacknz May 26, 2023 12:21

add permissions to selected list of datastreams

62ce182

gsantoro requested review from a team as code owners May 26, 2023 12:25

gsantoro requested review from gizas and tetianakravchenko May 26, 2023 12:25

gsantoro added draft Draft and removed draft Draft labels May 26, 2023

update PR id

c162ee9

gsantoro added 2 commits May 26, 2023 14:45

Merge branch 'main' into feature/allow_routing

4368286

fix merge conflict with main

1f4e99a

tommyers-elastic approved these changes May 26, 2023

View reviewed changes

gizas reviewed May 26, 2023

View reviewed changes

packages/activemq/changelog.yml Outdated Show resolved Hide resolved

gizas approved these changes May 26, 2023

View reviewed changes

gsantoro added 5 commits June 12, 2023 10:31

Merge branch 'main' into feature/allow_routing

6595db6

added comments for new settings

bb9b2fa

update activemq version

cadccb7

Merge branch 'main' into feature/allow_routing

70e0f90

Merge branch 'main' into feature/allow_routing

22eb873

gsantoro merged commit 835c3c7 into elastic:main Jun 13, 2023

gsantoro deleted the feature/allow_routing branch June 13, 2023 13:29

felixbarny linked an issue Jun 13, 2023 that may be closed by this pull request

Allow routing for integrations that are not input packages #6255

Closed

felixbarny mentioned this pull request Jun 13, 2023

Allow routing for integrations that are not input packages #6255

Closed

felixbarny mentioned this pull request Jun 14, 2023

Enable routing data to different data streams elastic/apm-server#10991

Closed

ruflin reviewed Jun 15, 2023

View reviewed changes

P1llus mentioned this pull request Oct 25, 2023

[UDP/TCP] Add reroute processor support #8294

Merged

4 tasks

andrewkroh added Integration:activemq ActiveMQ Integration:auditd Auditd Logs Integration:aws AWS Integration:docker Docker Integration:kafka Kafka Integration:kubernetes Kubernetes Integration:system System labels Jul 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow routing for integrations that are not input packages #6340

Allow routing for integrations that are not input packages #6340

gsantoro commented May 26, 2023

elasticmachine commented May 26, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented May 26, 2023 •

edited

Loading

tommyers-elastic left a comment •

edited

Loading

gsantoro commented May 26, 2023

gsantoro commented Jun 13, 2023

gsantoro commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

ruflin Jun 15, 2023

ishleenk17 Jun 28, 2023 •

edited

Loading

lalit-satapathy Jun 28, 2023

Allow routing for integrations that are not input packages #6340

Allow routing for integrations that are not input packages #6340

Conversation

gsantoro commented May 26, 2023

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented May 26, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented May 26, 2023 • edited Loading

🌐 Coverage report

tommyers-elastic left a comment • edited Loading

Choose a reason for hiding this comment

gsantoro commented May 26, 2023

gsantoro commented Jun 13, 2023

gsantoro commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

💚 Build Succeeded

History

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

ruflin Jun 15, 2023

Choose a reason for hiding this comment

ishleenk17 Jun 28, 2023 • edited Loading

Choose a reason for hiding this comment

lalit-satapathy Jun 28, 2023

Choose a reason for hiding this comment

elasticmachine commented May 26, 2023 •

edited

Loading

elasticmachine commented May 26, 2023 •

edited

Loading

tommyers-elastic left a comment •

edited

Loading

ishleenk17 Jun 28, 2023 •

edited

Loading