aws_s3 source should support codecs

[jszwedko]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Users want to use AWS S3 as a transport mechanism between Vector instances. The aws_s3 source currently only supports logs, so this doesn't work for metrics (or traces).

Attempted Solutions

No response

Proposal

Add codec options to the aws_s3 source to enable the use of native or native_json codecs to send data through AWS S3 as a transport.

References

I think usage of codecs could be introduced somewhere around here:

vector/src/sources/aws_s3/mod.rs

Lines 291 to 306 in c59d71c

	use Compression::*;
	match compression {
	Auto => unreachable!(), // is mapped above
	None => Box::new(r),
	Gzip => Box::new({
	let mut decoder = bufread::GzipDecoder::new(r);
	decoder.multiple_members(true);
	decoder
	}),
	Zstd => Box::new({
	let mut decoder = bufread::ZstdDecoder::new(r);
	decoder.multiple_members(true);
	decoder
	}),
	}
	}

The config option should be added to

vector/src/sources/aws_s3/mod.rs

Line 74 in c59d71c

pub struct AwsS3Config {

Example configuration:

vector/src/sources/http_client/client.rs

Lines 70 to 78 in c59d71c

	/// Decoder to use on the HTTP responses.
	#[configurable(derived)]
	#[serde(default = "default_decoding")]
	pub decoding: DeserializerConfig,
	/// Framing to use in the decoding.
	#[configurable(derived)]
	#[serde(default = "default_framing_message_based")]
	pub framing: FramingConfig,

That source can be used as a good reference for applying those codec options too.

Version

vector 0.28.1

[jszwedko]

Closed by #17098