Add AWSHealth Integration #10119

agithomas · 2024-06-11T06:50:17Z

Type of change

Enhancement

Proposed commit message

Create AWS Health Integration.
AWS health metrics provide insights into the health of your AWS environment by monitoring various aspects, such as open issues, scheduled maintenance events, etc.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

elastic-package build
elastic-package stack up -v -d --services package-registry
Configure the AWS Health Integration.
Verify the dashboard data
Validate using Discover interface

Related issues

Screenshots

agithomas · 2024-07-06T02:50:49Z

The PR is blocked because of the issue : #10400

agithomas · 2024-07-07T08:55:33Z

/test

elasticmachine · 2024-07-07T10:26:51Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

packages/aws/data_stream/awshealth/elasticsearch/ingest_pipeline/default.yml

ishleenk17 · 2024-07-08T03:41:11Z

packages/aws/data_stream/awshealth/fields/ecs.yml

+  name: cloud.account.id
+  dimension: true
+- external: ecs
+  name: cloud.region


Please follow the ecs@mappings template.

Removed non-dimension fields from ecs.yml file

packages/aws/data_stream/awshealth/fields/fields.yml

packages/aws/data_stream/awshealth/agent/stream/stream.yml.hbs

packages/aws/data_stream/awshealth/sample_event.json

packages/aws/data_stream/awshealth/fields/fields.yml

muthu-mps · 2024-07-08T04:56:35Z

The metrics include the status_code which is more useful for monitoring. Can we include a visualisation which breaks down metrics by status_code.

packages/aws/data_stream/awshealth/fields/fields.yml

packages/aws/data_stream/awshealth/manifest.yml

packages/aws/data_stream/awshealth/fields/fields.yml

packages/aws/data_stream/awshealth/fields/package-fields.yml

zmoog

@agithomas, should we also set up the .github/CODEOWNERS file for this integration?

agithomas · 2024-07-09T02:15:11Z

The metrics include the status_code which is more useful for monitoring. Can we include a visualisation which breaks down metrics by status_code.

This already exists. Snippet from the screenshot. The metrics are broken down into Status & Regionwise. Status + Region would be more usable and hence this approach is followed

…gs, regenerated sample_events

agithomas · 2024-07-09T03:58:47Z

@agithomas, should we also set up the .github/CODEOWNERS file for this integration?

Thanks for pointing it out. Added the entry.

packages/aws/_dev/build/docs/awshealth.md

packages/aws/docs/awshealth.md

muthu-mps

LGTM!

shmsr

Also, see this patch:

Makes small adjustments to make all descriptions consistent.

diff --git a/packages/aws/data_stream/awshealth/fields/fields.yml b/packages/aws/data_stream/awshealth/fields/fields.yml
index 14795aa42..260f40da1 100644
--- a/packages/aws/data_stream/awshealth/fields/fields.yml
+++ b/packages/aws/data_stream/awshealth/fields/fields.yml
@@ -4,7 +4,7 @@
     - name: awshealth
       type: group
       description: >
-        AWS Health metrics
+        AWS Health metrics.
       fields:
         - name: affected_entities_others
           type: float
@@ -33,12 +33,12 @@
           type: keyword
           dimension: true
           description: >
-            The unique identifier for the event. The event ARN has the format `arn:aws:health:<event-region>::event/<SERVICE>/<EVENT_TYPE_CODE>/<EVENT_TYPE_PLUS_ID>`. For example, `arn:aws:health:us-east-1::event/EC2/EC2_INSTANCE_RETIREMENT_SCHEDULED/EC2_INSTANCE_RETIREMENT_SCHEDULED_ABC123-DEF456`
+            The unique identifier for the event. The event ARN has the format `arn:aws:health:<event-region>::event/<SERVICE>/<EVENT_TYPE_CODE>/<EVENT_TYPE_PLUS_ID>`. For example, `arn:aws:health:us-east-1::event/EC2/EC2_INSTANCE_RETIREMENT_SCHEDULED/EC2_INSTANCE_RETIREMENT_SCHEDULED_ABC123-DEF456`.
 
         - name: event_scope_code
           type: keyword
           description: >
-            This parameter specifies whether the Health event is a public Amazon Web Service event or an account-specific event. Allowed values are `PUBLIC`, `ACCOUNT_SPECIFIC`, or `NONE`."
+            The parameter that specifies whether the Health event is a public Amazon Web Service event or an account-specific event. Allowed values are `PUBLIC`, `ACCOUNT_SPECIFIC`, or `NONE`.
 
         - name: event_type_category
           type: keyword
@@ -85,7 +85,7 @@
           object_type: keyword
           object_type_mapping_type: "*"
           description: >
-            Details of the affected entities related to the event.
+            The details of the affected entities related to the event.
 
         - name: affected_entities.aws_account_id
           type: keyword
@@ -105,7 +105,7 @@
         - name: affected_entities.last_updated_time
           type: date
           description: >
-            The last updated time of entity.
+            The last updated time of the entity.
 
         - name: affected_entities.status_code
           type: keyword

I have a couple of suggestions for package-fields.yml but from our earlier discussion, you want to keep it as it in data_streams for consistency; so skipping those nitpicks.

shmsr · 2024-07-09T10:37:30Z

packages/aws/data_stream/awshealth/manifest.yml

+        multi: true
+        required: false
+        show_user: true
+      - name: tags_filter


Add description for tags_filter, region, period and data_granularity.

shmsr · 2024-07-09T10:45:06Z

Just a couple of comments. Rest looks good; ready to approve when these comments get addressed.

agithomas · 2024-07-11T07:13:53Z

Makes small adjustments to make all descriptions consistent.

Addressed the suggestions.

zmoog

LGTM!

I only left a comment to remind us that we may need to revisit the definitions of the data_stream.* and @timestamp fields in the future.

zmoog · 2024-07-11T08:46:44Z

packages/aws/data_stream/awshealth/fields/base-fields.yml

+- name: data_stream.type
+  type: constant_keyword
+  description: Data stream type.
+- name: data_stream.dataset
+  type: constant_keyword
+  description: Data stream dataset.
+- name: data_stream.namespace
+  type: constant_keyword
+  description: Data stream namespace.
+- name: '@timestamp'
+  type: date
+  description: Event timestamp.


IIRC, ECS includes definitions for the data_stream.* and @timestamp fields, so we may not need to include these ones.

Base
https://www.elastic.co/guide/en/ecs/current/ecs-base.html

Data Stream
https://www.elastic.co/guide/en/ecs/current/ecs-data_stream.html

However, I just noticed that elastic-package explicitly checks for these fields, so removing them is not an option until we update elastic-package.

I'll leave this comment as a reminder we may need to remove these field definitions in the future.

@zmoog : Please see this

elastic-sonarqube · 2024-07-11T10:26:35Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
86.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-07-11T10:26:41Z

💚 Build Succeeded

Buildkite Build
Commit: b9bf787

History

💚 Build #13518 succeeded 17317da
💚 Build #13413 succeeded 21b93e6
💚 Build #13396 succeeded 5172fb5
💚 Build #13388 succeeded 5685031
💚 Build #13346 succeeded 9551350
💚 Build #13341 succeeded 7c544fd

elasticmachine · 2024-07-12T05:28:23Z

Package aws - 2.18.0 containing this change is available at https://epr.elastic.co/search?package=aws

agithomas added 2 commits June 11, 2024 06:49

Add AWSHealth Integration

7762350

Added PR Link

1417dd1

agithomas mentioned this pull request Jun 12, 2024

[AWS] Create AWS Health Integration package #10111

Closed

Merge branch 'main' into issue-10111-awshealth

7c544fd

agithomas mentioned this pull request Jul 6, 2024

[AWS][WAF] System test failing for WAF with 8.14.0 #10400

Closed

agithomas marked this pull request as ready for review July 7, 2024 11:34

agithomas requested review from a team as code owners July 7, 2024 11:34

gpop63 reviewed Jul 7, 2024

View reviewed changes

packages/aws/data_stream/awshealth/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

Added better null handling script

9551350

ishleenk17 reviewed Jul 8, 2024

View reviewed changes

muthu-mps reviewed Jul 8, 2024

View reviewed changes

packages/aws/data_stream/awshealth/sample_event.json Outdated Show resolved Hide resolved

packages/aws/data_stream/awshealth/fields/fields.yml Outdated Show resolved Hide resolved

packages/aws/data_stream/awshealth/fields/fields.yml Outdated Show resolved Hide resolved

muthu-mps reviewed Jul 8, 2024

View reviewed changes

packages/aws/data_stream/awshealth/fields/fields.yml Outdated Show resolved Hide resolved

shmsr requested changes Jul 8, 2024

View reviewed changes

zmoog reviewed Jul 8, 2024

View reviewed changes

Added codeowners entry, added processor in manifest, fixed ecs mappin…

5685031

…gs, regenerated sample_events

muthu-mps reviewed Jul 9, 2024

View reviewed changes

packages/aws/_dev/build/docs/awshealth.md Outdated Show resolved Hide resolved

Removed unwanted lines from the README

5172fb5

shmsr added the enhancement New feature or request label Jul 9, 2024

dej611 approved these changes Jul 9, 2024

View reviewed changes

muthu-mps reviewed Jul 9, 2024

View reviewed changes

packages/aws/docs/awshealth.md Show resolved Hide resolved

Fixed documentation issue

21b93e6

muthu-mps approved these changes Jul 9, 2024

View reviewed changes

shmsr requested changes Jul 9, 2024

View reviewed changes

Doc improvements added. Added missing auth settings

17317da

shmsr approved these changes Jul 11, 2024

View reviewed changes

zmoog approved these changes Jul 11, 2024

View reviewed changes

Removed latency settings

b9bf787

gpop63 approved these changes Jul 11, 2024

View reviewed changes

agithomas merged commit d4dcb0a into elastic:main Jul 12, 2024
5 checks passed

andrewkroh added the Integration:aws AWS label Jul 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add AWSHealth Integration #10119

Add AWSHealth Integration #10119

agithomas commented Jun 11, 2024 •

edited by shmsr

Loading

agithomas commented Jul 6, 2024

agithomas commented Jul 7, 2024

elasticmachine commented Jul 7, 2024 •

edited

Loading

ishleenk17 Jul 8, 2024

agithomas Jul 9, 2024

muthu-mps commented Jul 8, 2024

zmoog left a comment

agithomas commented Jul 9, 2024

agithomas commented Jul 9, 2024

muthu-mps left a comment

shmsr left a comment

shmsr Jul 9, 2024

shmsr commented Jul 9, 2024

agithomas commented Jul 11, 2024

zmoog left a comment

zmoog Jul 11, 2024

ishleenk17 Jul 11, 2024

elastic-sonarqube bot commented Jul 11, 2024

elasticmachine commented Jul 11, 2024

elasticmachine commented Jul 12, 2024

Add AWSHealth Integration #10119

Add AWSHealth Integration #10119

Conversation

agithomas commented Jun 11, 2024 • edited by shmsr Loading

Type of change

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

agithomas commented Jul 6, 2024

agithomas commented Jul 7, 2024

elasticmachine commented Jul 7, 2024 • edited Loading

🚀 Benchmarks report

ishleenk17 Jul 8, 2024

Choose a reason for hiding this comment

agithomas Jul 9, 2024

Choose a reason for hiding this comment

muthu-mps commented Jul 8, 2024

zmoog left a comment

Choose a reason for hiding this comment

agithomas commented Jul 9, 2024

agithomas commented Jul 9, 2024

muthu-mps left a comment

Choose a reason for hiding this comment

shmsr left a comment

Choose a reason for hiding this comment

shmsr Jul 9, 2024

Choose a reason for hiding this comment

shmsr commented Jul 9, 2024

agithomas commented Jul 11, 2024

zmoog left a comment

Choose a reason for hiding this comment

zmoog Jul 11, 2024

Choose a reason for hiding this comment

ishleenk17 Jul 11, 2024

Choose a reason for hiding this comment

elastic-sonarqube bot commented Jul 11, 2024

Quality Gate passed

elasticmachine commented Jul 11, 2024

💚 Build Succeeded

History

elasticmachine commented Jul 12, 2024

agithomas commented Jun 11, 2024 •

edited by shmsr

Loading

elasticmachine commented Jul 7, 2024 •

edited

Loading