Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AWS] Change SQS metrics statistic method #8521

Merged
merged 4 commits into from
Nov 20, 2023
Merged

[AWS] Change SQS metrics statistic method #8521

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
64b8110
Change SQS metrics statistic method
kaiyan-sheng Nov 15, 2023
17c4cb2
add changelog
kaiyan-sheng Nov 15, 2023
67793dd
Merge branch 'main' into sqs_stats
kaiyan-sheng Nov 15, 2023
3faa1cf
Update changelog.yml with more details
kaiyan-sheng Nov 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/aws/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "2.9.1"
Copy link
Member

@shmsr shmsr Nov 16, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Came here from the related SDH.

It appears that changes made in the PR warrant either a minor or a major version upgrade, instead of a patch upgrade.

Changing statistic method for a few metrics could have implications for backward compatibility, making it more of a breaking change. Therefore, a major upgrade would be more appropriate. What do you think?

Ref: https://semver.org/#summary

Additionally, +1 to Tom's comment.

changes:
- description: Change SQS metrics statistic method, which includes changing ApproximateAgeOfOldestMessage from average to max, changing NumberOfMessagesDeleted, NumberOfEmptyReceives, NumberOfMessagesReceived and NumberOfMessagesSent from average to sum.
type: bugfix
link: https://github.com/elastic/integrations/pull/8521
- version: "2.9.0"
changes:
- description: Limit request tracer log count to five.
Expand Down
14 changes: 11 additions & 3 deletions packages/aws/data_stream/sqs/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,14 +42,22 @@ proxy_url: {{proxy_url}}
metrics:
- namespace: AWS/SQS
resource_type: sqs
statistic: ["Average"]
statistic: ["Maximum"]
name:
- ApproximateAgeOfOldestMessage
- namespace: AWS/SQS
resource_type: sqs
statistic: ["Average"]
name:
- ApproximateNumberOfMessagesDelayed
- ApproximateNumberOfMessagesNotVisible
- ApproximateNumberOfMessagesVisible
- SentMessageSize
- namespace: AWS/SQS
resource_type: sqs
statistic: ["Sum"]
name:
- NumberOfMessagesDeleted
- NumberOfEmptyReceives
- NumberOfMessagesReceived
- NumberOfMessagesSent
- NumberOfEmptyReceives
- SentMessageSize
10 changes: 5 additions & 5 deletions packages/aws/data_stream/sqs/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ description: "Pipeline for SQS metrics"

processors:
- rename:
field: aws.sqs.metrics.ApproximateAgeOfOldestMessage.avg
field: aws.sqs.metrics.ApproximateAgeOfOldestMessage.max
target_field: aws.sqs.oldest_message_age.sec
ignore_missing: true
- rename:
Expand All @@ -19,19 +19,19 @@ processors:
target_field: aws.sqs.messages.visible
ignore_missing: true
- rename:
field: aws.sqs.metrics.NumberOfMessagesDeleted.avg
field: aws.sqs.metrics.NumberOfMessagesDeleted.sum
target_field: aws.sqs.messages.deleted
ignore_missing: true
- rename:
field: aws.sqs.metrics.NumberOfMessagesReceived.avg
field: aws.sqs.metrics.NumberOfMessagesReceived.sum
target_field: aws.sqs.messages.received
ignore_missing: true
- rename:
field: aws.sqs.metrics.NumberOfMessagesSent.avg
field: aws.sqs.metrics.NumberOfMessagesSent.sum
target_field: aws.sqs.messages.sent
ignore_missing: true
- rename:
field: aws.sqs.metrics.NumberOfEmptyReceives.avg
field: aws.sqs.metrics.NumberOfEmptyReceives.sum
target_field: aws.sqs.empty_receives
ignore_missing: true
- rename:
Expand Down
18 changes: 9 additions & 9 deletions packages/aws/data_stream/sqs/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,48 +16,48 @@
metric_type: gauge
format: duration
description: |
The approximate age of the oldest non-deleted message in the queue.
The maximum approximate age of the oldest non-deleted message in the queue.
- name: messages.delayed
type: long
metric_type: gauge
description: |
TThe number of messages in the queue that are delayed and not available for reading immediately.
The average number of messages in the queue that are delayed and not available for reading immediately.
- name: messages.not_visible
type: long
metric_type: gauge
description: |
The number of messages that are in flight.
The average number of messages that are in flight.
- name: messages.visible
type: long
metric_type: gauge
description: |
The number of messages available for retrieval from the queue.
The average number of messages available for retrieval from the queue.
- name: messages.deleted
type: long
metric_type: gauge
description: |
The number of messages deleted from the queue.
The total number of messages deleted from the queue.
- name: messages.received
type: long
metric_type: gauge
description: |
The number of messages returned by calls to the ReceiveMessage action.
The total number of messages returned by calls to the ReceiveMessage action.
- name: messages.sent
type: long
metric_type: gauge
description: |
The number of messages added to a queue.
The total number of messages added to a queue.
- name: empty_receives
type: long
metric_type: gauge
description: |
The number of ReceiveMessage API calls that did not return a message.
The total number of ReceiveMessage API calls that did not return a message.
- name: sent_message_size.bytes
type: long
metric_type: gauge
format: bytes
description: |
The size of messages added to a queue.
The average size of messages added to a queue.
- name: queue.name
type: keyword
description: |
Expand Down
18 changes: 9 additions & 9 deletions packages/aws/docs/sqs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,16 +128,16 @@ An example event for `sqs` looks as following:
| agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword | |
| aws.cloudwatch.namespace | The namespace specified when query cloudwatch api. | keyword | |
| aws.dimensions.QueueName | SQS queue name | keyword | |
| aws.sqs.empty_receives | The number of ReceiveMessage API calls that did not return a message. | long | gauge |
| aws.sqs.messages.delayed | TThe number of messages in the queue that are delayed and not available for reading immediately. | long | gauge |
| aws.sqs.messages.deleted | The number of messages deleted from the queue. | long | gauge |
| aws.sqs.messages.not_visible | The number of messages that are in flight. | long | gauge |
| aws.sqs.messages.received | The number of messages returned by calls to the ReceiveMessage action. | long | gauge |
| aws.sqs.messages.sent | The number of messages added to a queue. | long | gauge |
| aws.sqs.messages.visible | The number of messages available for retrieval from the queue. | long | gauge |
| aws.sqs.oldest_message_age.sec | The approximate age of the oldest non-deleted message in the queue. | long | gauge |
| aws.sqs.empty_receives | The total number of ReceiveMessage API calls that did not return a message. | long | gauge |
| aws.sqs.messages.delayed | The average number of messages in the queue that are delayed and not available for reading immediately. | long | gauge |
| aws.sqs.messages.deleted | The total number of messages deleted from the queue. | long | gauge |
| aws.sqs.messages.not_visible | The average number of messages that are in flight. | long | gauge |
| aws.sqs.messages.received | The total number of messages returned by calls to the ReceiveMessage action. | long | gauge |
| aws.sqs.messages.sent | The total number of messages added to a queue. | long | gauge |
| aws.sqs.messages.visible | The average number of messages available for retrieval from the queue. | long | gauge |
| aws.sqs.oldest_message_age.sec | The maximum approximate age of the oldest non-deleted message in the queue. | long | gauge |
| aws.sqs.queue.name | SQS queue name | keyword | |
| aws.sqs.sent_message_size.bytes | The size of messages added to a queue. | long | gauge |
| aws.sqs.sent_message_size.bytes | The average size of messages added to a queue. | long | gauge |
| aws.tags | Tag key value pairs from aws resources. | flattened | |
| cloud | Fields related to the cloud or infrastructure the events are coming from. | group | |
| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | |
Expand Down
2 changes: 1 addition & 1 deletion packages/aws/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 3.0.0
name: aws
title: AWS
version: 2.9.0
version: 2.9.1
description: Collect logs and metrics from Amazon Web Services (AWS) with Elastic Agent.
type: integration
categories:
Expand Down