Datadog Logs sink sacrifices 750KB on payload size for throughput and we'd like to avoid that sacrifice.

[blt]

In the Datadog logs sink we have to obey the constraint imposed by the Datadog Logs API that uncompressed, serialized payloads must not be larger than 5MB. The payloads are serialized as JSON and, while we know how large our in-memory Event instances are in bytes and, via the Batcher

vector/src/sinks/datadog/logs/sink.rs

Lines 280 to 292 in ee912a2

	let batcher = Batcher::new(
	input,
	EventPartitioner::default(),
	self.timeout,
	NonZeroUsize::new(MAX_PAYLOAD_ARRAY).unwrap(),
	NonZeroUsize::new(BATCH_GOAL_BYTES),
	)
	.map(|(maybe_key, batch)| {
	let key = maybe_key.unwrap_or_else(|| Arc::clone(&default_api_key));
	let request_builder = RequestBuilder::new(encoding.clone(), compression, log_schema);
	tokio::spawn(async move { request_builder.build(key, batch) })
	})
	.buffer_unordered(io_bandwidth);

know that we'll only ever move BATCH_GOAL_BYTES

vector/src/sinks/datadog/logs/sink.rs

Line 46 in ee912a2

const BATCH_GOAL_BYTES: usize = 4_250_000;

worth of Event instances down through our code we don't rightly know what the serialized size will be. To avoid making payloads that are too large we have set BATCH_GOAL_BYTES well below the 5MB limit.

We'd like to avoid that, if possible. 750KB is not nothing.

Previously this sink used a partitioning scheme for incoming Event instances that called to_raw_value on every incoming Event. Very useful because we knew exactly how large the JSON serialization was -- we had it -- but CPU intensive. We have considered solutions that would allow us to stream Events into a JSON writer but our implementation currently is based off serde_json which doesn't offer that kind of interface, with feedback on bytes written. We also have the slight difficulty that our tower based service stack here operates in a request/response fashion, meaning we can't easily stream Event instances up to some threshold and push back otherwise. A serde RawValue is likewise not minified, leaving some bytes on the table there as well. The Datadog Logs API accepts logs out-of-order so maintaining order in the sink is a non-goal.

What we would like to be able to achieve, all this said, are Datadog Logs payloads that are closer to 5MB without going over than we presently achieve without having lower throughput than the current method. Any solutions that we've come up with while shooting the breeze are essentially variants on bin-packing but with the serious limitation that we can't know the true size of an item to be binned without doing an expensive computation on it (the serialization).

[jszwedko]

Closing this since performance seems sufficient even with the sacrifice; we can let future profiling surface it again if needed.

[lukesteensen]

I was looking into this a bit today as part of #10020 and very quickly ran into the API limitation of 1000 events per payload. With that in mind, the 5MB limit really only comes into play if your log messages are approaching 5KB on average, which seems to me is quite a bit larger than would be common in the wild. We can certainly test with logs of that size, but it does make me question how big of an impact any work here would really have. It seems likely that we're hitting the event limit far more often than the byte limit.