Submit DBM query samples via new aggregator API #9045
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ghost
added
djova
added a commit
to DataDog/datadog-agent
that referenced
this pull request
Apr 9, 2021
Add a new aggregator API through which checks can submit "event platform events" of various types.
All supported `eventTypes` are hardcoded in `EventPlatformForwarder`.
The `dbm-samples` and `dbm-metrics` events are expected to arrive fully serialized so their pipelines are simply "HTTP passthrough" pipelines which skip all of the other features of logs pipelines like processing rules and encoding.
Future event types will be able to add more detailed processing if they need it.
**Overall flow**
1. `aggregator.submit_event_platform_event(check_id, rawEvent, "{eventType}")` - python API. Here's how the postgres check would be updated to use it: DataDog/integrations-core#9045.
2. `BufferedAggregator` forwards events to the `EventPlatformForwarder`. Events are **dropped** here if `EventPlatformForwarder` is backed up for any reason.
3. `EventPlatformForwarder` forwards events to the pipeline for the given `eventType`, **dropping** events for unknown `eventTypes`
**Internal Agent Stats**
*Prometheus*: `aggregator.flush - data_type:{eventType}, state:{ok|error}`
*ExpVar*: `EventPlatformEvents` & `EventPlatformEventsErrors`: counts by `eventType`
**User-Facing Agent Stats**
Statistics for each `eventType` will be tracked alongside other types of telemetry (`Service Checks`, `Series`, ...). Where appropriate, the raw `eventType` is translated to a human readable name (i.e. `dbm-samples` --> `Database Monitoring Query Samples`).
`agent status` output:
```
=========
Collector
=========
Running Checks
==============
postgres (5.4.0)
----------------
Instance ID: postgres:1df52d84fb6f603c [OK]
Metric Samples: Last Run: 366, Total: 7,527
Database Monitoring Query Samples: Last Run: 11, Total: 176
...
=========
Aggregator
=========
Checks Metric Sample: 29,818
Database Monitoring Query Samples: 473
...
```
`agent check {check_name}` output:
```
=== Metrics ===
...
=== Database Monitoring Query Samples ===
...
```
`agent check {check_name} --json` output will use the raw event types instead of the human readable names:
```
"aggregator": {
"metrics": [...],
"dbm-samples": [...],
...
}
```
**Motivation**
The posting of statement samples payloads to the intake for postgres & mysql checks is done directly from python as of (DataDog/integrations-core#8627, DataDog/integrations-core#8629). With this change we'll be able to move responsibility for posting payloads to the more robust agent go code with proper batching, buffering, retries, error handling, and tracking of statistics.
djova
changed the title
djova
force-pushed
the
djova/postgres-send-samples-through-aggregator
branch
from
e1e1f3f to
b60ba71
Compare
djova
force-pushed
the
djova/postgres-send-samples-through-aggregator
branch
2 times, most recently
from
165793d to
d2ac40b
Compare
ofek
reviewed
Apr 14, 2021
djova
added a commit
that referenced
this pull request
Apr 14, 2021
As of #9045 this client is unused and can be removed.
4 tasks
djova
force-pushed
the
djova/postgres-send-samples-through-aggregator
branch
from
acd080b to
04c9b3a
Compare
djova
added a commit
that referenced
this pull request
Apr 15, 2021
As of #9045 this client is unused and can be removed.
djova
force-pushed
the
djova/postgres-send-samples-through-aggregator
branch
from
04c9b3a to
6e00979
Compare
remeh
added a commit
to DataDog/datadog-agent
that referenced
this pull request
Apr 16, 2021
* add new generic event platform aggregator API
Add a new aggregator API through which checks can submit "event platform events" of various types.
All supported `eventTypes` are hardcoded in `EventPlatformForwarder`.
The `dbm-samples` and `dbm-metrics` events are expected to arrive fully serialized so their pipelines are simply "HTTP passthrough" pipelines which skip all of the other features of logs pipelines like processing rules and encoding.
Future event types will be able to add more detailed processing if they need it.
**Overall flow**
1. `aggregator.submit_event_platform_event(check_id, rawEvent, "{eventType}")` - python API. Here's how the postgres check would be updated to use it: DataDog/integrations-core#9045.
2. `BufferedAggregator` forwards events to the `EventPlatformForwarder`. Events are **dropped** here if `EventPlatformForwarder` is backed up for any reason.
3. `EventPlatformForwarder` forwards events to the pipeline for the given `eventType`, **dropping** events for unknown `eventTypes`
**Internal Agent Stats**
*Prometheus*: `aggregator.flush - data_type:{eventType}, state:{ok|error}`
*ExpVar*: `EventPlatformEvents` & `EventPlatformEventsErrors`: counts by `eventType`
**User-Facing Agent Stats**
Statistics for each `eventType` will be tracked alongside other types of telemetry (`Service Checks`, `Series`, ...). Where appropriate, the raw `eventType` is translated to a human readable name (i.e. `dbm-samples` --> `Database Monitoring Query Samples`).
`agent status` output:
```
=========
Collector
=========
Running Checks
==============
postgres (5.4.0)
----------------
Instance ID: postgres:1df52d84fb6f603c [OK]
Metric Samples: Last Run: 366, Total: 7,527
Database Monitoring Query Samples: Last Run: 11, Total: 176
...
=========
Aggregator
=========
Checks Metric Sample: 29,818
Database Monitoring Query Samples: 473
...
```
`agent check {check_name}` output:
```
=== Metrics ===
...
=== Database Monitoring Query Samples ===
...
```
`agent check {check_name} --json` output will use the raw event types instead of the human readable names:
```
"aggregator": {
"metrics": [...],
"dbm-samples": [...],
...
}
```
**Motivation**
The posting of statement samples payloads to the intake for postgres & mysql checks is done directly from python as of (DataDog/integrations-core#8627, DataDog/integrations-core#8629). With this change we'll be able to move responsibility for posting payloads to the more robust agent go code with proper batching, buffering, retries, error handling, and tracking of statistics.
* simplify
* remove debug log
* move json marshaling to check.go
* check enabled before lock
* refactor, add noop ep forwarder
* Update pkg/collector/check/stats.go
Co-authored-by: maxime mouial <[email protected]>
* remove purge during flush
* remove global
* Update rtloader/include/datadog_agent_rtloader.h
Co-authored-by: Rémy Mathieu <[email protected]>
* Update rtloader/common/builtins/aggregator.h
Co-authored-by: Rémy Mathieu <[email protected]>
* Update pkg/collector/check/stats.go
Co-authored-by: Rémy Mathieu <[email protected]>
* remove unnecessary
* rename lock
* refactor pipelines
* remove unnecessary nil check
* revert
* Update releasenotes/notes/event-platform-aggregator-api-33e92539f08ac5c2.yaml
Co-authored-by: Alexandre Yang <[email protected]>
* track processed
* move locking into ep forwarder
* move to top
* Update pkg/aggregator/aggregator.go
Co-authored-by: Alexandre Yang <[email protected]>
* remove read lock
* refactor error logging
* move to pkg/epforwarder
* update default dbm-metrics endpoint
* local var
Co-authored-by: maxime mouial <[email protected]>
Co-authored-by: Rémy Mathieu <[email protected]>
Co-authored-by: Alexandre Yang <[email protected]>
djova
force-pushed
the
djova/postgres-send-samples-through-aggregator
branch
from
6e00979 to
f1ee623
Compare
djova
force-pushed
the
djova/postgres-send-samples-through-aggregator
branch
2 times, most recently
from
8665e37 to
b7e4fdb
Compare
Follow-up to #9165: Update python & mysql checks to submit DBM events via the new aggregator API. Also improves the tests to better handle threads. Motivation: Submit events using the more robust agent go code to with proper batching, buffering, retries, error handling, and tracking of internal statistics.
djova
force-pushed
the
djova/postgres-send-samples-through-aggregator
branch
from
b7e4fdb to
888f71d
Compare
ofek
changed the title
ofek
approved these changes
Apr 16, 2021
djova
added a commit
that referenced
this pull request
Apr 19, 2021
As of #9045 this client is unused and can be removed.
djova
added a commit
that referenced
this pull request
Apr 28, 2021
As of #9045 this client is unused and can be removed.
djova
added a commit
that referenced
this pull request
May 7, 2021
As of #9045 this client is unused and can be removed.
djova
added a commit
that referenced
this pull request
May 24, 2021
As of #9045 this client is unused and can be removed.
ofek
pushed a commit
that referenced
this pull request
May 25, 2021
As of #9045 this client is unused and can be removed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Follow-up to #9165: Update python & mysql checks to submit DBM events via the new aggregator API.
Also improves the tests to better handle threads.
Motivation
Submit events using the more robust agent go code to with proper batching, buffering, retries, error handling, and tracking of internal statistics.
Review checklist (to be filled by reviewers)
changelog/andintegration/labels attached