Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable routing data to different data streams #10991

Closed
felixbarny opened this issue Jun 14, 2023 · 1 comment · Fixed by #11168
Closed

Enable routing data to different data streams #10991

felixbarny opened this issue Jun 14, 2023 · 1 comment · Fixed by #11168
Assignees
Milestone

Comments

@felixbarny
Copy link
Member

We'd like to allow users to use a reroute processor to send APM data to arbitrary namespaces. Currently, when using a Fleet-managed APM Server, sending data to a different data stream using the reroute process isn't possible. That's because the API Key permissions are very narrow and only allow sending data to an exact data stream.

Similarly to what we've done for some other integrations where we want to allow routing (see elastic/integrations#6340), we just need to add the elasticsearch.dynamic_dataset: true and elasticsearch.dynamic_namespace: true flags, accordingly.

I'm proposing to add elasticsearch.dynamic_namespace: true to all data streams and to add elasticsearch.dynamic_dataset: true only to the app_metrics and app_logs data streams. That's because changing the data set for other data stream, such as internal_metrics could confuse the APM UI. If required, we can always add elasticsearch.dynamic_dataset: true to other data streams, such as the traces data stream.

@felixbarny
Copy link
Member Author

One caveat for allowing routing to different datasets for custom metrics is that the target data streams need to have to have the dynamic templates and ingest processors that are used in the metrics-apm.app_metrics-* data stream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants