Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Render Mustache expressions in routing rules datasets values #1393

Merged
merged 10 commits into from
Aug 28, 2023
Merged

Render Mustache expressions in routing rules datasets values #1393

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
681e688
Render Mustache expressions in datasets value
zmoog Aug 14, 2023
b191cd0
Add tests for dataset expressions rendering
zmoog Aug 16, 2023
91cf7de
Switch to the github.com/cbroglie/mustache library
zmoog Aug 16, 2023
cfea2c4
Fix routing_rules.yml format
zmoog Aug 16, 2023
4a678ae
Remove the kubernetes test package
zmoog Aug 23, 2023
edd774a
Add a test rule with expression
zmoog Aug 23, 2023
9b9136c
Update expression tests
zmoog Aug 23, 2023
5c251cf
Update internal/fields/validate.go
zmoog Aug 25, 2023
ac18517
Make static checker happy
zmoog Aug 25, 2023
4db10c6
Make more realistic tests adding a `labels` field
zmoog Aug 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ require (
github.com/ProtonMail/gopenpgp/v2 v2.7.2
github.com/aymerick/raymond v2.0.2+incompatible
github.com/boumenot/gocover-cobertura v1.2.0
github.com/cbroglie/mustache v1.4.0
github.com/cespare/xxhash/v2 v2.2.0
github.com/dustin/go-humanize v1.0.1
github.com/elastic/elastic-integration-corpus-generator-tool v0.5.0
Expand Down
2 changes: 2 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ github.com/boumenot/gocover-cobertura v1.2.0 h1:g+VROIASoEHBrEilIyaCmgo7HGm+AV5y
github.com/boumenot/gocover-cobertura v1.2.0/go.mod h1:fz7ly8dslE42VRR5ZWLt2OHGDHjkTiA2oNvKgJEjLT0=
github.com/buger/jsonparser v1.1.1/go.mod h1:6RYKKt7H4d4+iWqouImQ9R2FZql3VbhNgx27UK13J/0=
github.com/bwesterb/go-ristretto v1.2.3/go.mod h1:fUIoIZaG73pV5biE2Blr2xEzDoMj7NFEuV9ekS419A0=
github.com/cbroglie/mustache v1.4.0 h1:Azg0dVhxTml5me+7PsZ7WPrQq1Gkf3WApcHMjMprYoU=
github.com/cbroglie/mustache v1.4.0/go.mod h1:SS1FTIghy0sjse4DUVGV1k/40B1qE1XkD9DtDsHo9iM=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44=
Expand Down
27 changes: 25 additions & 2 deletions internal/fields/validate.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ import (
"sort"
"strings"

"github.com/cbroglie/mustache"

zmoog marked this conversation as resolved.
Show resolved Hide resolved
"github.com/Masterminds/semver/v3"
"gopkg.in/yaml.v3"

Expand Down Expand Up @@ -316,12 +318,33 @@ func (v *Validator) validateDocumentValues(body common.MapStr) multierror.Error
if !v.specVersion.LessThan(semver2_0_0) && v.expectedDatasets != nil {
for _, datasetField := range datasetFieldNames {
value, err := body.GetValue(datasetField)
if err == common.ErrKeyNotFound {
if errors.Is(err, common.ErrKeyNotFound) {
continue
}

// Why do we render the expected datasets here?
// Because the expected datasets can contain
// mustache templates, and not just static
// strings.
//
// For example, the expected datasets for the
// Kubernetes container logs dataset can be:
//
// - "{{kubernetes.labels.elastic_co/dataset}}"
//
var renderedExpectedDatasets []string
for _, dataset := range v.expectedDatasets {
renderedDataset, err := mustache.Render(dataset, body)
if err != nil {
err := fmt.Errorf("can't render expected dataset %q: %w", dataset, err)
errs = append(errs, err)
return errs
}
renderedExpectedDatasets = append(renderedExpectedDatasets, renderedDataset)
}

str, ok := valueToString(value, v.disabledNormalization)
exists := stringInArray(str, v.expectedDatasets)
exists := stringInArray(str, renderedExpectedDatasets)
if !ok || !exists {
err := fmt.Errorf("field %q should have value in %q, it has \"%v\"",
datasetField, v.expectedDatasets, value)
Expand Down
38 changes: 38 additions & 0 deletions ...arallel/awsfirehose/data_stream/log/_dev/test/pipeline/test-cloudtrail-by-account-id.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{
"events": [
{
"message": "{\"eventVersion\":\"1.08\",\"userIdentity\":{\"type\":\"AWSService\",\"invokedBy\":\"cloudtrail.amazonaws.com\"},\"eventTime\":\"2023-07-17T21:02:26Z\",\"eventSource\":\"sts.amazonaws.com\",\"eventName\":\"AssumeRole\",\"awsRegion\":\"sa-east-1\",\"sourceIPAddress\":\"cloudtrail.amazonaws.com\",\"userAgent\":\"cloudtrail.amazonaws.com\",\"requestParameters\":{\"roleArn\":\"arn:aws:iam::123456:role/service-role/aws-cloudtrail-logs-123456-b888baff_Role\",\"roleSessionName\":\"CLOUDWATCH_LOGS_DELIVERY_SESSION\"},\"responseElements\":{\"credentials\":{\"accessKeyId\":\"ASIAZEDJODE3A5LVGLFB\",\"sessionToken\":\"IQoJb3JpZ2luX2VjEGUaCXNhLWVhc3QtMSJHMEUCIHgHmtcrhwDhosJlQVky+C2zsYDKuR99qVlNjGIp8FLWAiEAsJtTDQ3Arq8iXEOHwv0ImEQdGb5tbgc+fLpoK58Enb4q9AII3v//////////ARAEGgw2MjcyODYzNTAxMzQiDN5gNdfO4ZdSqDmmwSrIAicTBYZg+ZXjwiJTN/Bz2YsMWYU6psw5znG3/Gh3EJ1P3RCmB7d79X6XZzFVi2u2xdrnaY/sTKDfp1jdl8OoAsSKYwJiGbzjoQlv59bB6JqPbKfAKUPAmz6JEMWNFgWTtaQL9rNkdPz23u/1msoUSzxCcxR9f3A2dD4yqnVpNJe8ipuhxpBMzQ61vcGL4G5hQEDM/o8sORP2PXbK4O7QAuWOyuryYkHAPwY9RrL0WHfflGBEBQV6XlidGpsRCtIppZVn025n3DQOypDEaL3fKp0gUsMkDH+frFjxop4o4wRYC3CxXe3XRJ5/Te886rQry7RUfXlQtiCfojZO5ohcLB+z6Y/uCK0IHp3zrfl5shKsQIAFt7p0B8W7PK5yHE4W9HHRiktJ9wTtq1YCTaWECpnjW0bISNgumRmDOAJvVHAjSjfkr4yAlJkw4qm8pQY6vwGbBiuf98AfRFrXMy01hVdE3GNTBrIS68zxUJaOjBLgw8l0nEC00L+LPuqaASFWz65Dnq5JAjXaDD9E3iCi4klp4gZFAcj7uGgeBIPkP7Bpr4SvBfnnqCgE2oyFrWke3NnYtqkL5iHLJeGlOTrvI5ND2H4jurQv0KbiqwHt6DmGF3poZOrtf8R3piNcuCCDLU8RvhRVLHy5rKPzsWgNokBc9XXmgltwvB6rIgdZhBJzupzmy/NSoWZcOeH2ooEELw==\",\"expiration\":\"Jul 12, 2023, 10:02:26 PM\"},\"assumedRoleUser\":{\"assumedRoleId\":\"AROAZEDJODE3NLJAH2FZC:CLOUDWATCH_LOGS_DELIVERY_SESSION\",\"arn\":\"arn:aws:sts::123456:assumed-role/aws-cloudtrail-logs-123456-b888baff_Role/CLOUDWATCH_LOGS_DELIVERY_SESSION\"}},\"requestID\":\"041c9e5f-a031-47d2-a4a0-011bc8d5352c\",\"eventID\":\"3096b662-7aa9-43e6-8bee-541a45686745\",\"readOnly\":true,\"resources\":[{\"accountId\":\"123456\",\"type\":\"AWS::IAM::Role\",\"ARN\":\"arn:aws:iam::123456:role/service-role/aws-cloudtrail-logs-123456-b888baff_Role\"}],\"eventType\":\"AwsApiCall\",\"managementEvent\":true,\"recipientAccountId\":\"123456\",\"sharedEventID\":\"a1c94275-884f-4c1f-b8dc-2e1bf4c94d29\",\"eventCategory\":\"Management\"}",
"@timestamp": "2023-07-25T21:04:35Z",
"aws": {
"firehose": {
"arn": "arn:aws:firehose:eu-west-1:123456:deliverystream/firehose-cloudtrail-logs-to-elastic",
"subscription_filters": "[cloudtrail-to-firehose]",
"request_id": "971ae05f-a128-4a7f-b623-30f9bc513e55"
},
"kinesis": {
"name": "firehose-cloudtrail-logs-to-elastic",
"type": "deliverystream"
},
"cloudwatch": {
"log_group": "aws-cloudtrail-logs-123456-1c167310",
"log_stream": "123456_CloudTrail_eu-west-1_3"
}
},
"cloud": {
"account": {
"id": "123456"
},
"region": "eu-west-1",
"provider": "aws"
},
"data_stream": {
"dataset": "awsfirehose.log",
"namespace": "default",
"type": "logs"
},
"event": {
"id": "37670326805251200781477669690942747782212394134076063744"
}
}
]
}
41 changes: 41 additions & 0 deletions ...ehose/data_stream/log/_dev/test/pipeline/test-cloudtrail-by-account-id.json-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
{
"expected": [
{
"@timestamp": "2023-07-25T21:04:35Z",
"aws": {
"cloudwatch": {
"log_group": "aws-cloudtrail-logs-123456-1c167310",
"log_stream": "123456_CloudTrail_eu-west-1_3"
},
"firehose": {
"arn": "arn:aws:firehose:eu-west-1:123456:deliverystream/firehose-cloudtrail-logs-to-elastic",
"request_id": "971ae05f-a128-4a7f-b623-30f9bc513e55",
"subscription_filters": "[cloudtrail-to-firehose]"
},
"kinesis": {
"name": "firehose-cloudtrail-logs-to-elastic",
"type": "deliverystream"
}
},
"cloud": {
"account": {
"id": "123456"
},
"provider": "aws",
"region": "eu-west-1"
},
"data_stream": {
"dataset": "123456",
"namespace": "123456",
"type": "logs"
},
"ecs": {
"version": "8.0.0"
},
"event": {
"id": "37670326805251200781477669690942747782212394134076063744"
},
"message": "{\"eventVersion\":\"1.08\",\"userIdentity\":{\"type\":\"AWSService\",\"invokedBy\":\"cloudtrail.amazonaws.com\"},\"eventTime\":\"2023-07-17T21:02:26Z\",\"eventSource\":\"sts.amazonaws.com\",\"eventName\":\"AssumeRole\",\"awsRegion\":\"sa-east-1\",\"sourceIPAddress\":\"cloudtrail.amazonaws.com\",\"userAgent\":\"cloudtrail.amazonaws.com\",\"requestParameters\":{\"roleArn\":\"arn:aws:iam::123456:role/service-role/aws-cloudtrail-logs-123456-b888baff_Role\",\"roleSessionName\":\"CLOUDWATCH_LOGS_DELIVERY_SESSION\"},\"responseElements\":{\"credentials\":{\"accessKeyId\":\"ASIAZEDJODE3A5LVGLFB\",\"sessionToken\":\"IQoJb3JpZ2luX2VjEGUaCXNhLWVhc3QtMSJHMEUCIHgHmtcrhwDhosJlQVky+C2zsYDKuR99qVlNjGIp8FLWAiEAsJtTDQ3Arq8iXEOHwv0ImEQdGb5tbgc+fLpoK58Enb4q9AII3v//////////ARAEGgw2MjcyODYzNTAxMzQiDN5gNdfO4ZdSqDmmwSrIAicTBYZg+ZXjwiJTN/Bz2YsMWYU6psw5znG3/Gh3EJ1P3RCmB7d79X6XZzFVi2u2xdrnaY/sTKDfp1jdl8OoAsSKYwJiGbzjoQlv59bB6JqPbKfAKUPAmz6JEMWNFgWTtaQL9rNkdPz23u/1msoUSzxCcxR9f3A2dD4yqnVpNJe8ipuhxpBMzQ61vcGL4G5hQEDM/o8sORP2PXbK4O7QAuWOyuryYkHAPwY9RrL0WHfflGBEBQV6XlidGpsRCtIppZVn025n3DQOypDEaL3fKp0gUsMkDH+frFjxop4o4wRYC3CxXe3XRJ5/Te886rQry7RUfXlQtiCfojZO5ohcLB+z6Y/uCK0IHp3zrfl5shKsQIAFt7p0B8W7PK5yHE4W9HHRiktJ9wTtq1YCTaWECpnjW0bISNgumRmDOAJvVHAjSjfkr4yAlJkw4qm8pQY6vwGbBiuf98AfRFrXMy01hVdE3GNTBrIS68zxUJaOjBLgw8l0nEC00L+LPuqaASFWz65Dnq5JAjXaDD9E3iCi4klp4gZFAcj7uGgeBIPkP7Bpr4SvBfnnqCgE2oyFrWke3NnYtqkL5iHLJeGlOTrvI5ND2H4jurQv0KbiqwHt6DmGF3poZOrtf8R3piNcuCCDLU8RvhRVLHy5rKPzsWgNokBc9XXmgltwvB6rIgdZhBJzupzmy/NSoWZcOeH2ooEELw==\",\"expiration\":\"Jul 12, 2023, 10:02:26 PM\"},\"assumedRoleUser\":{\"assumedRoleId\":\"AROAZEDJODE3NLJAH2FZC:CLOUDWATCH_LOGS_DELIVERY_SESSION\",\"arn\":\"arn:aws:sts::123456:assumed-role/aws-cloudtrail-logs-123456-b888baff_Role/CLOUDWATCH_LOGS_DELIVERY_SESSION\"}},\"requestID\":\"041c9e5f-a031-47d2-a4a0-011bc8d5352c\",\"eventID\":\"3096b662-7aa9-43e6-8bee-541a45686745\",\"readOnly\":true,\"resources\":[{\"accountId\":\"123456\",\"type\":\"AWS::IAM::Role\",\"ARN\":\"arn:aws:iam::123456:role/service-role/aws-cloudtrail-logs-123456-b888baff_Role\"}],\"eventType\":\"AwsApiCall\",\"managementEvent\":true,\"recipientAccountId\":\"123456\",\"sharedEventID\":\"a1c94275-884f-4c1f-b8dc-2e1bf4c94d29\",\"eventCategory\":\"Management\"}"
}
]
}
5 changes: 5 additions & 0 deletions test/packages/parallel/awsfirehose/data_stream/log/routing_rules.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
- source_dataset: awsfirehose.log
rules:
- target_dataset:
- "{{cloud.account.id}}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For a test this looks enough, but isn't this intended to be used with fields that contain datasets? Like {{data_stream.dataset}} or similar?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, expressions in the routing rules enable the rule to pick a dataset or namespace value for a field in the log event.

For example, the Kubernetes integration picks the dataset value from the {{kubernetes.labels.elastic_co/dataset}} field.

Unfortunately, the Firehose event does not offer a lot of fields with values compatible with the data stream naming restrictions.

So, I imagined a scenario where users want to partition the log events into multiple AWS account-based data streams using {{cloud.account.id}}. But it's a bit confusing.

The other option is to add an extra field in the log event, for example, {{labels.elastic_co/dataset}}, and use this field in the rule. It does not exist in the Firehose realm but is less confusing overall. WDYT?

namespace:
- "{{cloud.account.id}}"
if: ctx.cloud?.region == 'eu-west-1'
- target_dataset: aws.cloudtrail
if: ctx['aws.cloudwatch.log_stream'].contains('CloudTrail')
namespace:
Expand Down