Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fleet] Warn/inform user when they elect to send data to an existing data stream #149427

Closed
hop-dev opened this issue Jan 24, 2023 · 2 comments · Fixed by #151075
Closed

[Fleet] Warn/inform user when they elect to send data to an existing data stream #149427

hop-dev opened this issue Jan 24, 2023 · 2 comments · Fixed by #151075
Assignees
@hop-dev
Labels
estimate:small Small Estimated Level of Effort Team:Fleet Team label for Observability Data Collection Fleet team

Comments

@hop-dev
Copy link
Contributor

hop-dev commented Jan 24, 2023

Part of #133296

Now that #145529 is complete, we create index templates, component templates and ingest pipelines at package policy creation time for the dataset specified by the user.

If the user elects to send data to an existing data stream that is not managed by the current package, we do not create the index template etc. On the API, we reject a request to send the data to an existing datastream unless the force flag is provided, the UI transparently adds this flag for the user. This action is very easy to do and pretty much encouraged by the UI, however there is a caveat which should be conveyed to the user:

No mappings or settings defined by the package will be applied to the destination data stream, so there is no guarantee the data generated by the integration willl "match" the destination index. these mappings and settings are defined by the fields in the integration and also any custom mappings and settings defined in the integration manifest elasticsearch key.

We should probably add a warning to the GUI and a short doc to link to?
@hop-dev hop-dev added the Team:Fleet Team label for Observability Data Collection Fleet team label Jan 24, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@hop-dev hop-dev mentioned this issue Jan 24, 2023
Closed
12 tasks
@joshdover
Copy link
Contributor

+1 to warning the user when this is encountered.

We also need to handle this same situation but in the opposite order: user first creates an input policy that ships to nginx.access and then later installs the nginx package. What happens? We probably want package installation would fail with an error indicating which policy is blocking this.

@hop-dev hop-dev added the estimate:small Small Estimated Level of Effort label Jan 31, 2023
@hop-dev hop-dev mentioned this issue Feb 13, 2023
Merged
hop-dev added a commit that referenced this issue Feb 14, 2023
@hop-dev
[Fleet] Warn user when they are sending data to stream managed by dif…
25d4c9f 
…ferent integration (#151075)

## Summary

Closes #149427

Give the user a warning if they are sending data to a data stream that
isn't managed by the integration being added.

<img width="697" alt="Screenshot 2023-02-13 at 21 17 13"
src="https://user-images.githubusercontent.com/3315046/218576500-a0817207-451e-47c5-ba0b-d0faaddca1a4.png">
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Feb 14, 2023
@hop-dev
[Fleet] Warn user when they are sending data to stream managed by dif…
dc2aeb0 
…ferent integration (elastic#151075)

## Summary

Closes elastic#149427

Give the user a warning if they are sending data to a data stream that
isn't managed by the integration being added.

<img width="697" alt="Screenshot 2023-02-13 at 21 17 13"
src="https://user-images.githubusercontent.com/3315046/218576500-a0817207-451e-47c5-ba0b-d0faaddca1a4.png">

(cherry picked from commit 25d4c9f)
kibanamachine added a commit that referenced this issue Feb 14, 2023
@kibanamachine @hop-dev
[8.7] [Fleet] Warn user when they are sending data to stream managed …
ff2877b 
…by different integration (#151075) (#151146)

# Backport

This will backport the following commits from `main` to `8.7`:
- [[Fleet] Warn user when they are sending data to stream managed by
different integration
(#151075)](#151075)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Mark
Hopkin","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-02-14T14:40:46Z","message":"[Fleet]
Warn user when they are sending data to stream managed by different
integration (#151075)\n\n## Summary\r\n\r\nCloses #149427\r\n\r\nGive
the user a warning if they are sending data to a data stream
that\r\nisn't managed by the integration being added.\r\n\r\n<img
width=\"697\" alt=\"Screenshot 2023-02-13 at 21 17
13\"\r\nsrc=\"https://user-images.githubusercontent.com/3315046/218576500-a0817207-451e-47c5-ba0b-d0faaddca1a4.png\">","sha":"25d4c9f3297bfcf685287d4b0fd3a91934d8eb04","branchLabelMapping":{"^v8.8.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Fleet","v8.7.0","v8.8.0"],"number":151075,"url":"https://github.com/elastic/kibana/pull/151075","mergeCommit":{"message":"[Fleet]
Warn user when they are sending data to stream managed by different
integration (#151075)\n\n## Summary\r\n\r\nCloses #149427\r\n\r\nGive
the user a warning if they are sending data to a data stream
that\r\nisn't managed by the integration being added.\r\n\r\n<img
width=\"697\" alt=\"Screenshot 2023-02-13 at 21 17
13\"\r\nsrc=\"https://user-images.githubusercontent.com/3315046/218576500-a0817207-451e-47c5-ba0b-d0faaddca1a4.png\">","sha":"25d4c9f3297bfcf685287d4b0fd3a91934d8eb04"}},"sourceBranch":"main","suggestedTargetBranches":["8.7"],"targetPullRequestStates":[{"branch":"8.7","label":"v8.7.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.8.0","labelRegex":"^v8.8.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/151075","number":151075,"mergeCommit":{"message":"[Fleet]
Warn user when they are sending data to stream managed by different
integration (#151075)\n\n## Summary\r\n\r\nCloses #149427\r\n\r\nGive
the user a warning if they are sending data to a data stream
that\r\nisn't managed by the integration being added.\r\n\r\n<img
width=\"697\" alt=\"Screenshot 2023-02-13 at 21 17
13\"\r\nsrc=\"https://user-images.githubusercontent.com/3315046/218576500-a0817207-451e-47c5-ba0b-d0faaddca1a4.png\">","sha":"25d4c9f3297bfcf685287d4b0fd3a91934d8eb04"}}]}]
BACKPORT-->

Co-authored-by: Mark Hopkin <[email protected]>
justinkambic pushed a commit to justinkambic/kibana that referenced this issue Feb 23, 2023
@hop-dev @justinkambic
[Fleet] Warn user when they are sending data to stream managed by dif…
85bc6c2 
…ferent integration (elastic#151075)

## Summary

Closes elastic#149427

Give the user a warning if they are sending data to a data stream that
isn't managed by the integration being added.

<img width="697" alt="Screenshot 2023-02-13 at 21 17 13"
src="https://user-images.githubusercontent.com/3315046/218576500-a0817207-451e-47c5-ba0b-d0faaddca1a4.png">
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hop-dev hop-dev

Labels
estimate:small Small Estimated Level of Effort Team:Fleet Team label for Observability Data Collection Fleet team
Projects
None yet
Milestone
No milestone
3 participants
@joshdover @hop-dev @elasticmachine