-
Notifications
You must be signed in to change notification settings - Fork 92
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: compress kept trace decision message #1430
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good! I had a couple of comments, but they're optional.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good. I'd also like to see compression for drop traces as it'll help log term.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's try it!
Which problem is this PR solving?
Due to the multiplication effect of pubsub messages through Redis, we would like to bring down the amount of network traffic sent by Redis.
One of the biggest contributors is
TraceDecison
message used to communicate kept trace decisions.This PR uses
snappy
to compress the data and then encoded usinggob
Here's the benchmark results comparing to the original JSON encoding.
In the
BenchmarkCompressionSizes
test, I generated a batch of 1000 trace decisions as the input. The amount of decrease in data size is 90%I also used
sync.Pool
for both compression and decompression buffers as well assnappy.Writer
to reduce allocation