[Ingest Manager] Refuse invalid stream values in configuration #19587
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,184 @@ | ||
| // Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| // or more contributor license agreements. Licensed under the Elastic License; | ||
| // you may not use this file except in compliance with the Elastic License. | ||
|
|
||
| package filters | ||
|
|
||
| import ( | ||
| "strings" | ||
|
|
||
| "github.com/elastic/beats/v7/x-pack/elastic-agent/pkg/agent/errors" | ||
| "github.com/elastic/beats/v7/x-pack/elastic-agent/pkg/agent/transpiler" | ||
| "github.com/elastic/beats/v7/x-pack/elastic-agent/pkg/core/logger" | ||
| ) | ||
|
|
||
| // ErrInvalidNamespace is error returned when namespace value provided is invalid. | ||
| var ErrInvalidNamespace = errors.New("provided namespace is invalid", errors.TypeConfig) | ||
|
|
||
| // ErrInvalidDataset is error returned when dataset name value provided is invalid. | ||
| var ErrInvalidDataset = errors.New("provided dataset name is invalid", errors.TypeConfig) | ||
|
|
||
| // StreamChecker checks for invalid values in stream namespace and dataset. | ||
| func StreamChecker(log *logger.Logger, ast *transpiler.AST) error { | ||
| inputsNode, found := transpiler.Lookup(ast, "inputs") | ||
| if !found { | ||
| return nil | ||
| } | ||
|
|
||
| inputsNodeList, ok := inputsNode.Value().(*transpiler.List) | ||
| if !ok { | ||
| return nil | ||
| } | ||
|
|
||
| inputsNodeListCollection, ok := inputsNodeList.Value().([]transpiler.Node) | ||
| if !ok { | ||
| return errors.New("inputs is not a list", errors.TypeConfig) | ||
| } | ||
|
|
||
| for _, inputNode := range inputsNodeListCollection { | ||
| // fail only if dataset.namespace or dataset[namespace] is found and invalid | ||
| // not provided values are ok and will be fixed by rules | ||
| if nsNode, found := inputNode.Find("dataset.namespace"); found { | ||
| nsKey, ok := nsNode.(*transpiler.Key) | ||
| if ok { | ||
| if newNamespace := nsKey.Value().(transpiler.Node).String(); !isNamespaceValid(newNamespace) { | ||
| return ErrInvalidNamespace | ||
| } | ||
| } | ||
| } else { | ||
| dsNode, found := inputNode.Find("dataset") | ||
| if found { | ||
| // got a dataset | ||
| datasetMap, ok := dsNode.Value().(*transpiler.Dict) | ||
| if ok { | ||
| nsNode, found := datasetMap.Find("namespace") | ||
| if found { | ||
| nsKey, ok := nsNode.(*transpiler.Key) | ||
| if ok { | ||
| if newNamespace := nsKey.Value().(transpiler.Node).String(); !isNamespaceValid(newNamespace) { | ||
| return ErrInvalidNamespace | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| streamsNode, ok := inputNode.Find("streams") | ||
| if !ok { | ||
| continue | ||
| } | ||
|
|
||
| streamsList, ok := streamsNode.Value().(*transpiler.List) | ||
| if !ok { | ||
| continue | ||
| } | ||
|
|
||
| streamNodes, ok := streamsList.Value().([]transpiler.Node) | ||
| if !ok { | ||
| return errors.New("streams is not a list", errors.TypeConfig) | ||
|
There was a problem hiding this comment. Have you tested with Endpoint Security enabled in Fleet? I think this will cause an issue, because it does come with https://github.com/elastic/beats/pull/19248/files#diff-68f0d6baed417771ae64bf9e5a7587a7R267 There was a problem hiding this comment. good point here |
||
| } | ||
|
|
||
| for _, streamNode := range streamNodes { | ||
| streamMap, ok := streamNode.(*transpiler.Dict) | ||
| if !ok { | ||
| continue | ||
| } | ||
|
|
||
| // fix this only if in compact form | ||
| if dsNameNode, found := streamMap.Find("dataset.name"); found { | ||
| dsKey, ok := dsNameNode.(*transpiler.Key) | ||
| if ok { | ||
| if newDataset := dsKey.Value().(transpiler.Node).String(); !isDatasetValid(newDataset) { | ||
| return ErrInvalidDataset | ||
| } | ||
| } | ||
| } else { | ||
| datasetNode, found := streamMap.Find("dataset") | ||
| if found { | ||
| datasetMap, ok := datasetNode.Value().(*transpiler.Dict) | ||
| if !ok { | ||
| continue | ||
| } | ||
|
|
||
| dsNameNode, found := datasetMap.Find("name") | ||
| if found { | ||
| dsKey, ok := dsNameNode.(*transpiler.Key) | ||
| if ok { | ||
| if newDataset := dsKey.Value().(transpiler.Node).String(); !isDatasetValid(newDataset) { | ||
| return ErrInvalidDataset | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| return nil | ||
| } | ||
|
|
||
| // The only two requirement are that it has only characters allowed in an Elasticsearch index name | ||
| // and does NOT contain a `-`. | ||
| // Index names must meet the following criteria: | ||
| // Lowercase only | ||
| // Cannot include \, /, *, ?, ", <, >, |, ` ` (space character), ,, # | ||
| // Cannot start with -, _, + | ||
| // Cannot be . or .. | ||
|
|
||
| func isNamespaceValid(namespace string) bool { | ||
| // Cannot be . or .. | ||
| if namespace == "." || namespace == ".." { | ||
| return false | ||
| } | ||
|
|
||
| if len(namespace) <= 0 || len(namespace) > 255 { | ||
| return false | ||
| } | ||
|
|
||
| // Lowercase only | ||
| if strings.ToLower(namespace) != namespace { | ||
| return false | ||
| } | ||
|
|
||
| // Cannot include \, /, *, ?, ", <, >, |, ` ` (space character), ,, # | ||
| if strings.ContainsAny(namespace, "\\/*?\"<>| ,#-") { | ||
| return false | ||
| } | ||
|
|
||
| // Cannot start with -, _, + | ||
| if strings.HasPrefix(namespace, "-") || strings.HasPrefix(namespace, "_") || strings.HasPrefix(namespace, "+") { | ||
| return false | ||
| } | ||
|
|
||
| return true | ||
| } | ||
|
|
||
| // The same requirements as for the namespace apply. | ||
| func isDatasetValid(dataset string) bool { | ||
| // Cannot be . or .. | ||
| if dataset == "." || dataset == ".." { | ||
| return false | ||
| } | ||
|
|
||
| if len(dataset) <= 0 || len(dataset) > 255 { | ||
| return false | ||
| } | ||
|
|
||
| // Lowercase only | ||
| if strings.ToLower(dataset) != dataset { | ||
| return false | ||
| } | ||
|
|
||
| // Cannot include \, /, *, ?, ", <, >, |, ` ` (space character), ,, # | ||
| if strings.ContainsAny(dataset, "\\/*?\"<>| ,#-") { | ||
| return false | ||
| } | ||
|
|
||
| // Cannot start with -, _, + | ||
| if strings.HasPrefix(dataset, "-") || strings.HasPrefix(dataset, "_") || strings.HasPrefix(dataset, "+") { | ||
| return false | ||
| } | ||
|
|
||
| return true | ||
|
There was a problem hiding this comment. Why duplicate the same function and just give it the same name? Could remove the second one and rename it to That way in the future if the validation of dataset does differ from namespace, then only that function body needs to be changed. |
||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it would be better to be explicit on what is allowed even if it is conservative in nature. That way its less likely for them to make a mistake and its clear directly from the comment.
How about:
Must be all lowercase, no spaces, only characters a-z or numbers 0-9 (no special characters), and no more than 255 characters.I think the character length might be an issue being its concat-ed to make the index name.