Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for AWS Access Point ARN in compatible integrations #12154

Merged
merged 8 commits into from
Jan 3, 2025
Merged

Add support for AWS Access Point ARN in compatible integrations #12154

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
ded493e
Add support for access_point_arn setting in compatible integrations
chemamartinez Dec 18, 2024
88e5383
Update changelog
chemamartinez Dec 18, 2024
f151292
Merge branch 'main' into 11737-access_point_arn-awss3
chemamartinez Dec 24, 2024
5ceacdc
Add Shourie's suggestions
chemamartinez Dec 24, 2024
9eb2ef1
Apply suggestions in setting descriptions
chemamartinez Jan 2, 2025
693f66c
Merge branch 'main' into 11737-access_point_arn-awss3
chemamartinez Jan 2, 2025
443f479
Merge branch 'main' into 11737-access_point_arn-awss3
chemamartinez Jan 3, 2025
11aa305
Fix Sentinel One Cloud Funnel pipeline tests
chemamartinez Jan 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/amazon_security_lake/_dev/build/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ The Amazon Security Lake integration collects logs from both [Third-party servic
![Role ARN and External ID](../img/role_arn_and_external_id.png)

6. If you want to collect logs via AWS S3, then you have to put the following details:
- bucket arn
- bucket ARN or access point ARN
- role ARN
- external id

Expand Down
5 changes: 5 additions & 0 deletions packages/amazon_security_lake/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "2.3.0"
changes:
- description: Add support for Access Point ARN when collecting logs via the AWS S3 Bucket.
type: enhancement
link: https://github.com/elastic/integrations/pull/12154
- version: "2.2.0"
changes:
- description: Add "preserve_original_event" tag to documents with `event.kind` manually set to "pipeline_error".
Expand Down
2 changes: 2 additions & 0 deletions packages/amazon_security_lake/data_stream/event/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

{{#if bucket_arn}}
bucket_arn: {{bucket_arn}}
{{else if access_point_arn}}
access_point_arn: {{access_point_arn}}
{{/if}}
{{#if number_of_workers}}
number_of_workers: {{number_of_workers}}
Expand Down
9 changes: 8 additions & 1 deletion packages/amazon_security_lake/data_stream/event/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,14 @@ streams:
multi: false
required: false
show_user: true
description: ARN of the AWS S3 bucket that will be polled for list operation. It is a required parameter for collecting logs via the AWS S3 Bucket.
description: ARN of the AWS S3 bucket that will be polled for list operation. It is a required parameter for collecting logs via the AWS S3 Bucket unless you set an Access Point ARN.
- name: access_point_arn
type: text
title: "[S3] Access Point ARN"
multi: false
required: false
show_user: true
description: ARN of the AWS S3 Access Point that will be polled for list operation. This is an alternative to the Bucket ARN, one of them is required for collecting logs via the AWS S3 Bucket.
- name: bucket_list_prefix
type: text
title: "[S3] Bucket Prefix"
Expand Down
2 changes: 1 addition & 1 deletion packages/amazon_security_lake/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ The Amazon Security Lake integration collects logs from both [Third-party servic
![Role ARN and External ID](../img/role_arn_and_external_id.png)

6. If you want to collect logs via AWS S3, then you have to put the following details:
- bucket arn
- bucket ARN or access point ARN
- role ARN
- external id

Expand Down
4 changes: 2 additions & 2 deletions packages/amazon_security_lake/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
format_version: "3.0.3"
name: amazon_security_lake
title: Amazon Security Lake
version: "2.2.0"
version: "2.3.0"
description: Collect logs from Amazon Security Lake with Elastic Agent.
type: integration
categories: ["aws", "security"]
conditions:
kibana:
version: "^8.13.0"
version: "^8.16.2"
elastic:
subscription: basic
screenshots:
Expand Down
5 changes: 5 additions & 0 deletions packages/aws/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "2.37.0"
changes:
- description: Add support for Access Point ARN when collecting logs via the AWS S3 Bucket.
type: enhancement
link: https://github.com/elastic/integrations/pull/12154
- version: "2.36.2"
changes:
- description: Include pipeline test examples to accommodate the new Cloudtrail format for the `CreateGroup` and `UpdateGroup` event types.
Expand Down
2 changes: 2 additions & 0 deletions packages/aws/data_stream/apigateway_logs/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

{{#if bucket_arn}}
bucket_arn: {{bucket_arn}}
{{else if access_point_arn}}
access_point_arn: {{access_point_arn}}
{{/if}}
{{#if number_of_workers}}
number_of_workers: {{number_of_workers}}
Expand Down
9 changes: 8 additions & 1 deletion packages/aws/data_stream/apigateway_logs/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,14 @@ streams:
multi: false
required: false
show_user: true
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket.
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket unless you set an Access Point ARN.
- name: access_point_arn
type: text
title: "[S3] Access Point ARN"
multi: false
required: false
show_user: true
description: This is an alternative to the Bucket ARN, one of them is required for collecting logs via the AWS S3 Bucket.
- name: queue_url
type: text
title: "[SQS] Queue URL"
Expand Down
2 changes: 2 additions & 0 deletions packages/aws/data_stream/cloudfront_logs/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

{{#if bucket_arn}}
bucket_arn: {{bucket_arn}}
{{else if access_point_arn}}
access_point_arn: {{access_point_arn}}
{{/if}}
{{#if number_of_workers}}
number_of_workers: {{number_of_workers}}
Expand Down
9 changes: 8 additions & 1 deletion packages/aws/data_stream/cloudfront_logs/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,14 @@ streams:
multi: false
required: false
show_user: true
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket.
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket unless you set an Access Point ARN.
- name: access_point_arn
type: text
title: "[S3] Access Point ARN"
multi: false
required: false
show_user: true
description: This is an alternative to the Bucket ARN, one of them is required for collecting logs via the AWS S3 Bucket.
- name: bucket_list_prefix
type: text
title: "[S3] Bucket Prefix"
Expand Down
2 changes: 2 additions & 0 deletions packages/aws/data_stream/cloudtrail/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

{{#if bucket_arn}}
bucket_arn: {{bucket_arn}}
{{else if access_point_arn}}
access_point_arn: {{access_point_arn}}
{{/if}}
{{#if number_of_workers}}
number_of_workers: {{number_of_workers}}
Expand Down
9 changes: 8 additions & 1 deletion packages/aws/data_stream/cloudtrail/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,14 @@ streams:
multi: false
required: false
show_user: true
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket.
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket unless you set an Access Point ARN.
- name: access_point_arn
type: text
title: "[S3] Access Point ARN"
multi: false
required: false
show_user: true
description: This is an alternative to the Bucket ARN, one of them is required for collecting logs via the AWS S3 Bucket.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nits: Suggestion to rephrase the description to clearly conveying the relationship with BucketARN

Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket unless you set a Bucket ARN.

Not sure, how important it is to mention - what happens if both BucketARN and AccessPoint ARNs are set, which one is be given higher preference? Would it be an important point to mention in the README ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestions.

Not sure, how important it is to mention - what happens if both BucketARN and AccessPoint ARNs are set, which one is be given higher preference? Would it be an important point to mention in the README ?

I think it is nice to have it. I added it at the description of Bucket ARN setting, as Readme don't talk about these particular settings in most of the integrations.

- name: queue_url
type: text
title: "[SQS] Queue URL"
Expand Down
2 changes: 2 additions & 0 deletions packages/aws/data_stream/ec2_logs/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

{{#if bucket_arn}}
bucket_arn: {{bucket_arn}}
{{else if access_point_arn}}
access_point_arn: {{access_point_arn}}
{{/if}}
{{#if number_of_workers}}
number_of_workers: {{number_of_workers}}
Expand Down
9 changes: 8 additions & 1 deletion packages/aws/data_stream/ec2_logs/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,14 @@ streams:
multi: false
required: false
show_user: true
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket.
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket unless you set an Access Point ARN.
- name: access_point_arn
type: text
title: "[S3] Access Point ARN"
multi: false
required: false
show_user: true
description: This is an alternative to the Bucket ARN, one of them is required for collecting logs via the AWS S3 Bucket.
- name: bucket_list_prefix
type: text
title: "[S3] Bucket Prefix"
Expand Down
2 changes: 2 additions & 0 deletions packages/aws/data_stream/elb_logs/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

{{#if bucket_arn}}
bucket_arn: {{bucket_arn}}
{{else if access_point_arn}}
access_point_arn: {{access_point_arn}}
{{/if}}
{{#if number_of_workers}}
number_of_workers: {{number_of_workers}}
Expand Down
9 changes: 8 additions & 1 deletion packages/aws/data_stream/elb_logs/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,14 @@ streams:
multi: false
required: false
show_user: true
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket.
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket unless you set an Access Point ARN.
- name: access_point_arn
type: text
title: "[S3] Access Point ARN"
multi: false
required: false
show_user: true
description: This is an alternative to the Bucket ARN, one of them is required for collecting logs via the AWS S3 Bucket.
- name: bucket_list_prefix
type: text
title: "[S3] Bucket Prefix"
Expand Down
2 changes: 2 additions & 0 deletions packages/aws/data_stream/emr_logs/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

{{#if bucket_arn}}
bucket_arn: {{bucket_arn}}
{{else if access_point_arn}}
access_point_arn: {{access_point_arn}}
{{/if}}
{{#if number_of_workers}}
number_of_workers: {{number_of_workers}}
Expand Down
9 changes: 8 additions & 1 deletion packages/aws/data_stream/emr_logs/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,14 @@ streams:
multi: false
required: false
show_user: true
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket.
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket unless you set an Access Point ARN.
- name: access_point_arn
type: text
title: "[S3] Access Point ARN"
multi: false
required: false
show_user: true
description: This is an alternative to the Bucket ARN, one of them is required for collecting logs via the AWS S3 Bucket.
- name: queue_url
type: text
title: "[SQS] Queue URL"
Expand Down
2 changes: 2 additions & 0 deletions packages/aws/data_stream/firewall_logs/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

{{#if bucket_arn}}
bucket_arn: {{bucket_arn}}
{{else if access_point_arn}}
access_point_arn: {{access_point_arn}}
{{/if}}
{{#if number_of_workers}}
number_of_workers: {{number_of_workers}}
Expand Down
9 changes: 8 additions & 1 deletion packages/aws/data_stream/firewall_logs/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,14 @@ streams:
multi: false
required: false
show_user: true
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket.
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket unless you set an Access Point ARN.
- name: access_point_arn
type: text
title: "[S3] Access Point ARN"
multi: false
required: false
show_user: true
description: This is an alternative to the Bucket ARN, one of them is required for collecting logs via the AWS S3 Bucket.
- name: bucket_list_prefix
type: text
title: "[S3] Bucket Prefix"
Expand Down
2 changes: 2 additions & 0 deletions packages/aws/data_stream/guardduty/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

{{#if bucket_arn}}
bucket_arn: {{bucket_arn}}
{{else if access_point_arn}}
access_point_arn: {{access_point_arn}}
{{/if}}
{{#if number_of_workers}}
number_of_workers: {{number_of_workers}}
Expand Down
9 changes: 8 additions & 1 deletion packages/aws/data_stream/guardduty/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,14 @@ streams:
multi: false
required: false
show_user: true
description: It is a required parameter for collecting logs via the AWS S3 Bucket.
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket unless you set an Access Point ARN.
- name: access_point_arn
type: text
title: "[S3] Access Point ARN"
multi: false
required: false
show_user: true
description: This is an alternative to the Bucket ARN, one of them is required for collecting logs via the AWS S3 Bucket.
- name: bucket_list_prefix
type: text
title: "[S3] Bucket Prefix"
Expand Down
2 changes: 2 additions & 0 deletions packages/aws/data_stream/route53_resolver_logs/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

{{#if bucket_arn}}
bucket_arn: {{bucket_arn}}
{{else if access_point_arn}}
access_point_arn: {{access_point_arn}}
{{/if}}
{{#if number_of_workers}}
number_of_workers: {{number_of_workers}}
Expand Down
9 changes: 8 additions & 1 deletion packages/aws/data_stream/route53_resolver_logs/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,14 @@ streams:
multi: false
required: false
show_user: true
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket.
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket unless you set an Access Point ARN.
- name: access_point_arn
type: text
title: "[S3] Access Point ARN"
multi: false
required: false
show_user: true
description: This is an alternative to the Bucket ARN, one of them is required for collecting logs via the AWS S3 Bucket.
- name: bucket_list_prefix
type: text
title: "[S3] Bucket Prefix"
Expand Down
2 changes: 2 additions & 0 deletions packages/aws/data_stream/s3access/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

{{#if bucket_arn}}
bucket_arn: {{bucket_arn}}
{{else if access_point_arn}}
access_point_arn: {{access_point_arn}}
{{/if}}
{{#if number_of_workers}}
number_of_workers: {{number_of_workers}}
Expand Down
9 changes: 8 additions & 1 deletion packages/aws/data_stream/s3access/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,14 @@ streams:
multi: false
required: false
show_user: true
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket.
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket unless you set an Access Point ARN.
- name: access_point_arn
type: text
title: "[S3] Access Point ARN"
multi: false
required: false
show_user: true
description: This is an alternative to the Bucket ARN, one of them is required for collecting logs via the AWS S3 Bucket.
- name: bucket_list_prefix
type: text
title: "[S3] Bucket Prefix"
Expand Down
2 changes: 2 additions & 0 deletions packages/aws/data_stream/vpcflow/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

{{#if bucket_arn}}
bucket_arn: {{bucket_arn}}
{{else if access_point_arn}}
access_point_arn: {{access_point_arn}}
{{/if}}
{{#if number_of_workers}}
number_of_workers: {{number_of_workers}}
Expand Down
9 changes: 8 additions & 1 deletion packages/aws/data_stream/vpcflow/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,14 @@ streams:
multi: false
required: false
show_user: true
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket.
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket unless you set an Access Point ARN.
- name: access_point_arn
type: text
title: "[S3] Access Point ARN"
multi: false
required: false
show_user: true
description: This is an alternative to the Bucket ARN, one of them is required for collecting logs via the AWS S3 Bucket.
- name: queue_url
type: text
title: "[SQS] Queue URL"
Expand Down
2 changes: 2 additions & 0 deletions packages/aws/data_stream/waf/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

{{#if bucket_arn}}
bucket_arn: {{bucket_arn}}
{{else if access_point_arn}}
access_point_arn: {{access_point_arn}}
{{/if}}
{{#if number_of_workers}}
number_of_workers: {{number_of_workers}}
Expand Down
9 changes: 8 additions & 1 deletion packages/aws/data_stream/waf/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,14 @@ streams:
multi: false
required: false
show_user: true
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket.
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket unless you set an Access Point ARN.
- name: access_point_arn
type: text
title: "[S3] Access Point ARN"
multi: false
required: false
show_user: true
description: This is an alternative to the Bucket ARN, one of them is required for collecting logs via the AWS S3 Bucket.
- name: bucket_list_prefix
type: text
title: "[S3] Bucket Prefix"
Expand Down
4 changes: 2 additions & 2 deletions packages/aws/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 3.0.0
name: aws
title: AWS
version: 2.36.2
version: 2.37.0
description: Collect logs and metrics from Amazon Web Services (AWS) with Elastic Agent.
type: integration
categories:
Expand All @@ -10,7 +10,7 @@ conditions:
elastic:
subscription: basic
kibana:
version: "^8.16.0"
version: "^8.16.2"
screenshots:
- src: /img/metricbeat-aws-overview.png
title: metricbeat aws overview
Expand Down
Loading