[AWS][Pipeline Test] Multiple pipeline Test failure due to event.category mappings

[agithomas]

Observing below mentioned failures upon running AWS pipeline test.
ES Stack Version : 8.14

This error breaks the CI/CD pipeline.

1

pipeline test: test-s3-server-access.log in aws.s3access
test case failed: one or more problems with fields found in documents: [0] field "event.category" is not normalized as expected: expected array, found "web" (string)

pipeline test: test-cloudfront.log in aws.cloudfront_logs
test case failed: one or more problems with fields found in documents: [0] field "event.category" is not normalized as expected: expected array, found "web" (string)

pipeline test: test-waf.log in aws.waf
test case failed: one or more problems with fields found in documents: [0] field "event.category" is not normalized as expected: expected array, found "web" (string)

Buildkite Reference : https://buildkite.com/elastic/integrations/builds/12418

The solution to the problem can be found here

https://github.com/elastic/elastic-package/blob/main/docs/howto/update_major_package_spec.md#field--is-not-normalized-as-expected-expected-array-found-

Once the issue is fixed, there may exist a need to run
elastic-package test pipeline -d cloudfront_logs -v --generate

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[agithomas]

With the above-mentioned changes applied, you can observe successful pipeline execution run.

ES version : 8.14

--- Test results for package: aws - START ---
╭─────────┬─────────────────┬───────────┬────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM     │ TEST TYPE │ TEST NAME                  │ RESULT │ TIME ELAPSED │
├─────────┼─────────────────┼───────────┼────────────────────────────┼────────┼──────────────┤
│ aws     │ cloudfront_logs │ pipeline  │ test-cloudfront.log        │ PASS   │  10.098706ms │
│ aws     │ cloudfront_logs │ pipeline  │ (ingest pipeline warnings) │ PASS   │ 571.062007ms │
╰─────────┴─────────────────┴───────────┴────────────────────────────┴────────┴──────────────╯

[agithomas]

For AWS-WAF, I have noticied below mentioned error additionally

--- Test results for package: aws - START ---
FAILURE DETAILS:
aws/waf test-waf.log:
[0] parsing field value failed: field "event.type" value "allowed" is not one of the expected values (access, error, info) for any of the values of "event.category" (web)
[1] parsing field value failed: field "event.type" value "denied" is not one of the expected values (access, error, info) for any of the values of "event.category" (web)

Fix Reference: https://github.com/elastic/elastic-package/blob/main/docs/howto/update_major_package_spec.md#field-eventtype-value--is-not-one-of-the-expected-values--for-

So, it may be determined what append processor values may be appropriate

But, I do notice the allowed value list: https://www.elastic.co/guide/en/ecs/current/ecs-allowed-values-event-type.html
which mentions values such as allowed and denied are supported.

[agithomas]

Closing the issue as the issue is not reproducible.

[agithomas]

Reopening the issue, as i could recreate the issue.

The mandatory condition to recreate the issue is

  kibana:
    version: "^8.14.0"
    

[agithomas]

@jsoriano , do you know why the issue mentioned here occur?

[muthu-mps]

@agithomas -

• By upgrading the stack version from 8.12 to 8.14 we may need to remove the ECS field definitions (ecs.yml file). We may also need to validate the ecs@mapping behaviour.
• Starting from 8.13 the ecs@mapping component template is referenced by any integration as mentioned here.

cc: @zmoog

[ishleenk17]

This issue will occur from 8.13 onwards, where the default ECS mappings are now applied to fields. Previously, event.category was not explicitly defined in ecs.yml, so it was treated as a keyword. The Elastic package did not previously flag such errors for undefined fields.

For your information, this upgrade of ECS mappings will be addressed as part of this issue. However, if this change is a prerequisite for your task, please feel free to make the necessary adjustments.

[agithomas]

Related issue : #10135

[jsoriano]

@jsoriano , do you know why the issue mentioned here occur?

As Ishleen mentions, I think this is a similar case to #9453 (comment). Summarizing, elastic-package ignores validation on some fields for legacy reasons when these fields are not defined, this includes event.*. This package didn't have definitions for some of these fields. Now that it supports only versions >= 8.13, it includes all ECS definitions, and these validations apply.

The possible solutions would be similar too:

• Adapt the values of event.type according to what ECS expects. But this could be a breaking change if there is something working with these values.
• Define event.type and/or event.category according to the expectations of this package, so the definition in ECS is not used for validation.

[agithomas]

Resolved by the PR