[Fleet] Allow to have custom pipeline at different level #170270
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nchaulet
added
release_note:skip
🤖 GitHub commentsExpand to view the GitHub comments
Just comment with:
|
nchaulet
force-pushed
the
feature-custom-pipeline
branch
from
323b0db to
fb111ef
Compare
|
Pinging @elastic/fleet (Team:Fleet) |
|
@kilfoyle we currently have a tutorial for |
kpollich
approved these changes
Nov 1, 2023
@nchaulet Thanks for letting me know about this. We could add a comment in the tutorial somewhere, but I think the most important spot would be this Ingest pipelines section on the "Data streams" page. It'd be really nice to give a simple example. |
|
I created elastic/ingest-docs#638 to capture the docs request here. |
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]
History
To update your PR or re-run it, just comment with: cc @nchaulet |
kibanamachine
added
v8.12.0
backport:skip
delanni
pushed a commit
to delanni/kibana
that referenced
this pull request
Nov 6, 2023
kpollich
added a commit
that referenced
this pull request
Jan 25, 2024
…ns + add descriptions to each pipeline (#175448) ## Summary Closes #175254 Ref #168019 Ref #170270 In 8.12.0, Fleet unintentionally shipped a breaking change in #170270 for APM users who make use of a custom `traces-apm` data stream. If a user had previously defined this ingest pipeline to customize documents ingested for the `traces-apm` data stream (defined [here](https://github.com/elastic/integrations/blob/9a36183f0bd12e39a957d2f7bd65f3de4ee685b1/packages/apm/data_stream/traces/manifest.yml#L2-L3), then they would unexpectedly see that pipeline called when documents were ingested to the `traces-apm.rum` and `traces-apm.sampled` datastreams as well. This PR addresses this collision by adding a `.package` suffix to the "package level" ingest pipeline introduced in 8.12.0. So, in 8.12.0 a processor would be defined as such on the `traces-apm.rum` or `traces-apm.sampled` ingest pipeline ``` { "pipeline": { "name": "traces-apm@custom", "ignore_missing_pipeline": true, } }, ``` This PR replaces the pipeline with one that looks as follows: ``` { "pipeline": { "name": "traces-apm.package@custom", "ignore_missing_pipeline": true, "description": "[Fleet] Pipeline for all data streams of type `traces` defined by the `apm` integration" } }, ``` **To be clear: this is a breaking change if you have defined the `traces-apm@custom` integration on 8.12. In 8.12.1, it will no longer be called for documents ingested to the `traces-apm`, `traces-apm.rum`, or `traces-apm.sampled` data streams. You will need to rename your pipeline to `traces-apm.package@custom` to preserve this behavior.** This change also applies to `logs-elastic_agent.*` ingest pipelines. See [this comment](#175254 (comment)) for more information. There is still technically room for a collision, though it's unlikely, if the data stream name is `package`. This will be handled by a package spec validation proposed in elastic/package-spec#699. --------- Co-authored-by: Kibana Machine <[email protected]>
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jan 25, 2024
…ns + add descriptions to each pipeline (elastic#175448) ## Summary Closes elastic#175254 Ref elastic#168019 Ref elastic#170270 In 8.12.0, Fleet unintentionally shipped a breaking change in elastic#170270 for APM users who make use of a custom `traces-apm` data stream. If a user had previously defined this ingest pipeline to customize documents ingested for the `traces-apm` data stream (defined [here](https://github.com/elastic/integrations/blob/9a36183f0bd12e39a957d2f7bd65f3de4ee685b1/packages/apm/data_stream/traces/manifest.yml#L2-L3), then they would unexpectedly see that pipeline called when documents were ingested to the `traces-apm.rum` and `traces-apm.sampled` datastreams as well. This PR addresses this collision by adding a `.package` suffix to the "package level" ingest pipeline introduced in 8.12.0. So, in 8.12.0 a processor would be defined as such on the `traces-apm.rum` or `traces-apm.sampled` ingest pipeline ``` { "pipeline": { "name": "traces-apm@custom", "ignore_missing_pipeline": true, } }, ``` This PR replaces the pipeline with one that looks as follows: ``` { "pipeline": { "name": "traces-apm.package@custom", "ignore_missing_pipeline": true, "description": "[Fleet] Pipeline for all data streams of type `traces` defined by the `apm` integration" } }, ``` **To be clear: this is a breaking change if you have defined the `traces-apm@custom` integration on 8.12. In 8.12.1, it will no longer be called for documents ingested to the `traces-apm`, `traces-apm.rum`, or `traces-apm.sampled` data streams. You will need to rename your pipeline to `traces-apm.package@custom` to preserve this behavior.** This change also applies to `logs-elastic_agent.*` ingest pipelines. See [this comment](elastic#175254 (comment)) for more information. There is still technically room for a collision, though it's unlikely, if the data stream name is `package`. This will be handled by a package spec validation proposed in elastic/package-spec#699. --------- Co-authored-by: Kibana Machine <[email protected]> (cherry picked from commit 9fe5a66)
kibanamachine
added a commit
that referenced
this pull request
Jan 25, 2024
…oid collisions + add descriptions to each pipeline (#175448) (#175547) # Backport This will backport the following commits from `main` to `8.12`: - [[Fleet] Update Fleet's custom ingest pipeline names to avoid collisions + add descriptions to each pipeline (#175448)](#175448) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Kyle Pollich","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-01-25T14:07:43Z","message":"[Fleet] Update Fleet's custom ingest pipeline names to avoid collisions + add descriptions to each pipeline (#175448)\n\n## Summary\r\n\r\nCloses https://github.com/elastic/kibana/issues/175254\r\nRef https://github.com/elastic/kibana/issues/168019\r\nRef https://github.com/elastic/kibana/pull/170270\r\n\r\nIn 8.12.0, Fleet unintentionally shipped a breaking change in\r\nhttps://github.com//pull/170270 for APM users who make use\r\nof a custom `traces-apm` data stream. If a user had previously defined\r\nthis ingest pipeline to customize documents ingested for the\r\n`traces-apm` data stream (defined\r\n[here](https://github.com/elastic/integrations/blob/9a36183f0bd12e39a957d2f7bd65f3de4ee685b1/packages/apm/data_stream/traces/manifest.yml#L2-L3),\r\nthen they would unexpectedly see that pipeline called when documents\r\nwere ingested to the `traces-apm.rum` and `traces-apm.sampled`\r\ndatastreams as well.\r\n\r\nThis PR addresses this collision by adding a `.package` suffix to the\r\n\"package level\" ingest pipeline introduced in 8.12.0.\r\n\r\nSo, in 8.12.0 a processor would be defined as such on the\r\n`traces-apm.rum` or `traces-apm.sampled` ingest pipeline\r\n\r\n```\r\n{\r\n \"pipeline\": {\r\n \"name\": \"traces-apm@custom\",\r\n \"ignore_missing_pipeline\": true,\r\n }\r\n},\r\n```\r\n\r\nThis PR replaces the pipeline with one that looks as follows:\r\n\r\n```\r\n{\r\n \"pipeline\": {\r\n \"name\": \"traces-apm.package@custom\",\r\n \"ignore_missing_pipeline\": true,\r\n \"description\": \"[Fleet] Pipeline for all data streams of type `traces` defined by the `apm` integration\"\r\n }\r\n},\r\n```\r\n\r\n**To be clear: this is a breaking change if you have defined the\r\n`traces-apm@custom` integration on 8.12. In 8.12.1, it will no longer be\r\ncalled for documents ingested to the `traces-apm`, `traces-apm.rum`, or\r\n`traces-apm.sampled` data streams. You will need to rename your pipeline\r\nto `traces-apm.package@custom` to preserve this behavior.**\r\n\r\nThis change also applies to `logs-elastic_agent.*` ingest pipelines. See\r\n[this\r\ncomment](https://github.com/elastic/kibana/issues/175254#issuecomment-1906202137)\r\nfor more information.\r\n\r\nThere is still technically room for a collision, though it's unlikely,\r\nif the data stream name is `package`. This will be handled by a package\r\nspec validation proposed in\r\nhttps://github.com/elastic/package-spec/issues/699.\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>","sha":"9fe5a66faf4e06fc444c6078edafc29e91126f8d","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:breaking","Team:Fleet","backport:prev-minor","v8.12.1","v8.13.0"],"title":"[Fleet] Update Fleet's custom ingest pipeline names to avoid collisions + add descriptions to each pipeline","number":175448,"url":"https://github.com/elastic/kibana/pull/175448","mergeCommit":{"message":"[Fleet] Update Fleet's custom ingest pipeline names to avoid collisions + add descriptions to each pipeline (#175448)\n\n## Summary\r\n\r\nCloses https://github.com/elastic/kibana/issues/175254\r\nRef https://github.com/elastic/kibana/issues/168019\r\nRef https://github.com/elastic/kibana/pull/170270\r\n\r\nIn 8.12.0, Fleet unintentionally shipped a breaking change in\r\nhttps://github.com//pull/170270 for APM users who make use\r\nof a custom `traces-apm` data stream. If a user had previously defined\r\nthis ingest pipeline to customize documents ingested for the\r\n`traces-apm` data stream (defined\r\n[here](https://github.com/elastic/integrations/blob/9a36183f0bd12e39a957d2f7bd65f3de4ee685b1/packages/apm/data_stream/traces/manifest.yml#L2-L3),\r\nthen they would unexpectedly see that pipeline called when documents\r\nwere ingested to the `traces-apm.rum` and `traces-apm.sampled`\r\ndatastreams as well.\r\n\r\nThis PR addresses this collision by adding a `.package` suffix to the\r\n\"package level\" ingest pipeline introduced in 8.12.0.\r\n\r\nSo, in 8.12.0 a processor would be defined as such on the\r\n`traces-apm.rum` or `traces-apm.sampled` ingest pipeline\r\n\r\n```\r\n{\r\n \"pipeline\": {\r\n \"name\": \"traces-apm@custom\",\r\n \"ignore_missing_pipeline\": true,\r\n }\r\n},\r\n```\r\n\r\nThis PR replaces the pipeline with one that looks as follows:\r\n\r\n```\r\n{\r\n \"pipeline\": {\r\n \"name\": \"traces-apm.package@custom\",\r\n \"ignore_missing_pipeline\": true,\r\n \"description\": \"[Fleet] Pipeline for all data streams of type `traces` defined by the `apm` integration\"\r\n }\r\n},\r\n```\r\n\r\n**To be clear: this is a breaking change if you have defined the\r\n`traces-apm@custom` integration on 8.12. In 8.12.1, it will no longer be\r\ncalled for documents ingested to the `traces-apm`, `traces-apm.rum`, or\r\n`traces-apm.sampled` data streams. You will need to rename your pipeline\r\nto `traces-apm.package@custom` to preserve this behavior.**\r\n\r\nThis change also applies to `logs-elastic_agent.*` ingest pipelines. See\r\n[this\r\ncomment](https://github.com/elastic/kibana/issues/175254#issuecomment-1906202137)\r\nfor more information.\r\n\r\nThere is still technically room for a collision, though it's unlikely,\r\nif the data stream name is `package`. This will be handled by a package\r\nspec validation proposed in\r\nhttps://github.com/elastic/package-spec/issues/699.\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>","sha":"9fe5a66faf4e06fc444c6078edafc29e91126f8d"}},"sourceBranch":"main","suggestedTargetBranches":["8.12"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","branchLabelMappingKey":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/175448","number":175448,"mergeCommit":{"message":"[Fleet] Update Fleet's custom ingest pipeline names to avoid collisions + add descriptions to each pipeline (#175448)\n\n## Summary\r\n\r\nCloses https://github.com/elastic/kibana/issues/175254\r\nRef https://github.com/elastic/kibana/issues/168019\r\nRef https://github.com/elastic/kibana/pull/170270\r\n\r\nIn 8.12.0, Fleet unintentionally shipped a breaking change in\r\nhttps://github.com//pull/170270 for APM users who make use\r\nof a custom `traces-apm` data stream. If a user had previously defined\r\nthis ingest pipeline to customize documents ingested for the\r\n`traces-apm` data stream (defined\r\n[here](https://github.com/elastic/integrations/blob/9a36183f0bd12e39a957d2f7bd65f3de4ee685b1/packages/apm/data_stream/traces/manifest.yml#L2-L3),\r\nthen they would unexpectedly see that pipeline called when documents\r\nwere ingested to the `traces-apm.rum` and `traces-apm.sampled`\r\ndatastreams as well.\r\n\r\nThis PR addresses this collision by adding a `.package` suffix to the\r\n\"package level\" ingest pipeline introduced in 8.12.0.\r\n\r\nSo, in 8.12.0 a processor would be defined as such on the\r\n`traces-apm.rum` or `traces-apm.sampled` ingest pipeline\r\n\r\n```\r\n{\r\n \"pipeline\": {\r\n \"name\": \"traces-apm@custom\",\r\n \"ignore_missing_pipeline\": true,\r\n }\r\n},\r\n```\r\n\r\nThis PR replaces the pipeline with one that looks as follows:\r\n\r\n```\r\n{\r\n \"pipeline\": {\r\n \"name\": \"traces-apm.package@custom\",\r\n \"ignore_missing_pipeline\": true,\r\n \"description\": \"[Fleet] Pipeline for all data streams of type `traces` defined by the `apm` integration\"\r\n }\r\n},\r\n```\r\n\r\n**To be clear: this is a breaking change if you have defined the\r\n`traces-apm@custom` integration on 8.12. In 8.12.1, it will no longer be\r\ncalled for documents ingested to the `traces-apm`, `traces-apm.rum`, or\r\n`traces-apm.sampled` data streams. You will need to rename your pipeline\r\nto `traces-apm.package@custom` to preserve this behavior.**\r\n\r\nThis change also applies to `logs-elastic_agent.*` ingest pipelines. See\r\n[this\r\ncomment](https://github.com/elastic/kibana/issues/175254#issuecomment-1906202137)\r\nfor more information.\r\n\r\nThere is still technically room for a collision, though it's unlikely,\r\nif the data stream name is `package`. This will be handled by a package\r\nspec validation proposed in\r\nhttps://github.com/elastic/package-spec/issues/699.\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>","sha":"9fe5a66faf4e06fc444c6078edafc29e91126f8d"}}]}] BACKPORT--> Co-authored-by: Kyle Pollich <[email protected]>
CoenWarmer
pushed a commit
to CoenWarmer/kibana
that referenced
this pull request
Feb 15, 2024
…ns + add descriptions to each pipeline (elastic#175448) ## Summary Closes elastic#175254 Ref elastic#168019 Ref elastic#170270 In 8.12.0, Fleet unintentionally shipped a breaking change in elastic#170270 for APM users who make use of a custom `traces-apm` data stream. If a user had previously defined this ingest pipeline to customize documents ingested for the `traces-apm` data stream (defined [here](https://github.com/elastic/integrations/blob/9a36183f0bd12e39a957d2f7bd65f3de4ee685b1/packages/apm/data_stream/traces/manifest.yml#L2-L3), then they would unexpectedly see that pipeline called when documents were ingested to the `traces-apm.rum` and `traces-apm.sampled` datastreams as well. This PR addresses this collision by adding a `.package` suffix to the "package level" ingest pipeline introduced in 8.12.0. So, in 8.12.0 a processor would be defined as such on the `traces-apm.rum` or `traces-apm.sampled` ingest pipeline ``` { "pipeline": { "name": "traces-apm@custom", "ignore_missing_pipeline": true, } }, ``` This PR replaces the pipeline with one that looks as follows: ``` { "pipeline": { "name": "traces-apm.package@custom", "ignore_missing_pipeline": true, "description": "[Fleet] Pipeline for all data streams of type `traces` defined by the `apm` integration" } }, ``` **To be clear: this is a breaking change if you have defined the `traces-apm@custom` integration on 8.12. In 8.12.1, it will no longer be called for documents ingested to the `traces-apm`, `traces-apm.rum`, or `traces-apm.sampled` data streams. You will need to rename your pipeline to `traces-apm.package@custom` to preserve this behavior.** This change also applies to `logs-elastic_agent.*` ingest pipelines. See [this comment](elastic#175254 (comment)) for more information. There is still technically room for a collision, though it's unlikely, if the data stream name is `package`. This will be handled by a package spec validation proposed in elastic/package-spec#699. --------- Co-authored-by: Kibana Machine <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Resolve #168019
Allow to have custom pipeline (
@custom) at different level instead of just${type}-${dataset}:global${type}${type}-${package}${type}-${dataset}This is done by adding a processor in the datastream default pipeline for each level.
Todo