Introduce a configurable sample rate #149
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hello 👋,
The projects where we use Spandex operate at a fairly large scale. A byproduct of this is that it becomes cost prohibitive to send all traces to Datadog. To solve this, we implemented our own Spandex sender that implements a sampling rate to determine trace priority before sending the trace to the Datadog agent.
This works pretty well, but falls apart when we use
continue_trace
to setup distributed tracing between different processes. Since priority is determined at the time of sending the trace, each trace priority is independently determined instead of the decision being made once and passed via trace context.To solve for this I've added an optional configuration for a sampling rate. This effectively enables head based sampling as described in the Datadog docs.