Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Azure] Migrate Azure logs package to adopt ecs@mappings #10224

Merged
merged 7 commits into from
Jul 8, 2024
Merged

[Azure] Migrate Azure logs package to adopt ecs@mappings #10224

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
0a68880
[azure] - change to ECS version [email protected]
harnish-elastic Jun 24, 2024
0fc5d53
update changelog.yml
harnish-elastic Jun 24, 2024
1ed4664
update readme
niraj-elastic Jun 24, 2024
b4138a6
update readme
niraj-elastic Jun 25, 2024
f452d01
add expected values to dns.header field in firewall_logs data stream
harnish-elastic Jun 26, 2024
3ba83d1
Updated "event.outcome" values from "Succeeded" to "success" and "Fai…
harnish-elastic Jul 5, 2024
8d2a674
Update packages/azure/changelog.yml
ishleenk17 Jul 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/azure/_dev/build/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
dependencies:
ecs:
reference: git@v8.0.0
reference: "git@v8.11.0"
4 changes: 4 additions & 0 deletions packages/azure/_dev/build/docs/activitylogs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,4 +64,8 @@ The `activitylogs` data stream of the Azure Logs package will collect any activi

{{event "activitylogs"}}

**ECS Field Reference**

Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.

{{fields "activitylogs"}}
16 changes: 16 additions & 0 deletions packages/azure/_dev/build/docs/adlogs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,10 @@ Retrieves Microsoft Entra ID sign-in logs. The sign-ins report provides informat

{{event "signinlogs"}}

**ECS Field Reference**

Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.

{{fields "signinlogs"}}

### Identity Protection logs
Expand All @@ -94,6 +98,10 @@ Retrieves Microsoft Entra ID Protection logs. The [Microsoft Entra ID Protection

{{event "identity_protection"}}

**ECS Field Reference**

Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.

{{fields "identity_protection"}}

### Provisioning logs
Expand All @@ -109,6 +117,10 @@ The Provisioning Logs contain a lot of details about a inbound/outbound sync act

{{event "provisioning"}}

**ECS Field Reference**

Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.

{{fields "provisioning"}}

### Audit logs
Expand All @@ -117,4 +129,8 @@ Retrieves Microsoft Entra ID audit logs. The audit logs provide traceability thr

{{event "auditlogs"}}

**ECS Field Reference**

Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.

{{fields "auditlogs"}}
4 changes: 4 additions & 0 deletions packages/azure/_dev/build/docs/application_gateway.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,4 +71,8 @@ The `application_gateway` data stream of the Azure Logs package will collect any

{{event "application_gateway"}}

**ECS Field Reference**

Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.

{{fields "application_gateway"}}
4 changes: 4 additions & 0 deletions packages/azure/_dev/build/docs/eventhub.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,4 +68,8 @@ The `eventhub` data stream of the Azure Logs package will collect any events tha

{{event "eventhub"}}

**ECS Field Reference**

Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.

{{fields "eventhub"}}
4 changes: 4 additions & 0 deletions packages/azure/_dev/build/docs/firewall_logs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,4 +72,8 @@ The `firewall_logs` data stream of the Azure Logs package will collect any firew

{{event "firewall_logs"}}

**ECS Field Reference**

Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.

{{fields "firewall_logs"}}
4 changes: 4 additions & 0 deletions packages/azure/_dev/build/docs/graphactivitylogs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,4 +81,8 @@ The `graphactivitylogs` data stream of the Azure Logs package will collect Micro

{{event "graphactivitylogs"}}

**ECS Field Reference**

Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.

{{fields "graphactivitylogs"}}
4 changes: 4 additions & 0 deletions packages/azure/_dev/build/docs/platformlogs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,4 +64,8 @@ The `platformlogs` dataset of the Azure Logs package will collect any platform e

{{event "platformlogs"}}

**ECS Field Reference**

Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.

{{fields "platformlogs"}}
4 changes: 4 additions & 0 deletions packages/azure/_dev/build/docs/springcloudlogs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,4 +66,8 @@ This is the `springcloudlogs` data stream of the Azure Logs package. It will col

{{event "springcloudlogs"}}

**ECS Field Reference**

Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.

{{fields "springcloudlogs"}}
8 changes: 8 additions & 0 deletions packages/azure/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,11 @@
- version: "1.12.0"
changes:
- description: ECS version updated to 8.11.0. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This changelog has multiple updates, Can we have separate descriptions for each.

  • Update ECS version to 8.11.0
  • Update the kibana constraint to ^8.13.0 to adopt ecs@mappings component template.

No need to add this sentence in the changelog entry,
Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the text that the automation writes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separate entries sounds like a reasonable enhancement to make to ecs-update. The related code is at https://github.com/andrewkroh/go-examples/blob/7bd2d3beac526b97b7734b2b51433d4be54d0fb6/ecs-update/ecs-update.go#L343 if anyone wants to contribute.

type: enhancement
link: https://github.com/elastic/integrations/pull/10224
- description: Updated "event.outcome" values from "Succeeded" to "success" and "Failed" to "failure".
type: breaking-change
link: https://github.com/elastic/integrations/pull/10224
- version: "1.11.4"
changes:
- description: Replace Azure AD with Microsoft Entra ID.
Expand Down
6 changes: 3 additions & 3 deletions ...data_stream/activitylogs/_dev/test/pipeline/test-activitylogs-edgecases.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
"provider": "azure"
},
"ecs": {
"version": "8.0.0"
"version": "8.11.0"
},
"event": {
"action": "Microsoft.Resourcehealth/healthevent/Updated/action",
Expand Down Expand Up @@ -261,7 +261,7 @@
"provider": "azure"
},
"ecs": {
"version": "8.0.0"
"version": "8.11.0"
},
"event": {
"action": "Sign-in activity",
Expand Down Expand Up @@ -518,7 +518,7 @@
"provider": "azure"
},
"ecs": {
"version": "8.0.0"
"version": "8.11.0"
},
"event": {
"action": "Sign-in activity",
Expand Down
2 changes: 1 addition & 1 deletion .../data_stream/activitylogs/_dev/test/pipeline/test-activitylogs-identity.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@
"provider": "azure"
},
"ecs": {
"version": "8.0.0"
"version": "8.11.0"
},
"event": {
"action": "Sign-in activity",
Expand Down
2 changes: 1 addition & 1 deletion ...azure/data_stream/activitylogs/_dev/test/pipeline/test-activitylogs-raw.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@
"provider": "azure"
},
"ecs": {
"version": "8.0.0"
"version": "8.11.0"
},
"event": {
"action": "MICROSOFT.EVENTHUB/NAMESPACES/AUTHORIZATIONRULES/LISTKEYS/ACTION",
Expand Down
2 changes: 1 addition & 1 deletion packages/azure/data_stream/activitylogs/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ description: Pipeline for parsing azure activity logs.
processors:
- set:
field: ecs.version
value: '8.0.0'
value: '8.11.0'
- rename:
field: azure
target_field: azure-eventhub
Expand Down
48 changes: 0 additions & 48 deletions packages/azure/data_stream/activitylogs/fields/agent.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,54 +1,6 @@
- name: cloud.account.id
external: ecs
- name: cloud.availability_zone
external: ecs
- name: cloud.instance.id
external: ecs
- name: cloud.instance.name
external: ecs
- name: cloud.machine.type
external: ecs
- name: cloud.provider
external: ecs
- name: cloud.region
external: ecs
- name: cloud.project.id
external: ecs
- name: cloud.image.id
type: keyword
description: Image ID for the cloud instance.
- name: container.id
external: ecs
- name: container.image.name
external: ecs
- name: container.labels
external: ecs
- name: container.name
external: ecs
- name: host.architecture
external: ecs
- name: host.domain
external: ecs
- name: host.hostname
external: ecs
- name: host.id
external: ecs
- name: host.mac
external: ecs
- name: host.name
external: ecs
- name: host.os.family
external: ecs
- name: host.os.kernel
external: ecs
- name: host.os.name
external: ecs
- name: host.os.platform
external: ecs
- name: host.os.version
external: ecs
- name: host.type
external: ecs
- name: host.containerized
type: boolean
description: If the host is a container.
Expand Down
96 changes: 0 additions & 96 deletions packages/azure/data_stream/activitylogs/fields/ecs.yml
View file
Open in desktop

This file was deleted.

Loading