Support setting an explicit prefix for the S3 store

[JamesBromberger]

Hello,
Can you please extend support for S3 by letting the user specify a prefix for the large objects to be put in S3. This means we don't have to dedicate a bucket explicitly for one service - especially when we're overlaying IAM policies for the access to the object store.

Current implementation is:
ExtendedClientConfiguration extendedClientConfig = new ExtendedClientConfiguration()
.withLargePayloadSupportEnabled(s3, s3BucketName);

Would be nice if we could do:
ExtendedClientConfiguration extendedClientConfig = new ExtendedClientConfiguration()
.withLargePayloadSupportEnabled(s3, s3BucketName, s3PrefixPattern);

Where pattern may contain:

• A complete string

It would be nice if this also supported expansion of some predefined variables, such as sprintf values:
%{y} - four digit year
%{m} - two digit month
%{d} - two digit day of month
%{aws::accountumber} - aws account number (in the case of a cross-account bucket being used)
%{aws::rolename} - IAM role name

Hence we can make prefix a string like:
"LARGE_MESSAGES/%{y}/%{m}/%{d}/%{aws::rolename}/"

I can then give the appropriate role the ability to CreateObject at that location, and any roles that read from it the ability to GetObject and DeleteObject (but not create or list).

[aws-aminsuzani]

Thanks for explaining your feature request for allowing the user to specify the prefix of S3 object keys. We will add your request to our backlog for future consideration. Also if you would like to submit a pull request, please go for it.

[glidester]

Hi, I've a similar need for organising the S3 objects with a meaningful prefix. I have implemented this in a fork here: glidester@eb75f07

Basically I've added the 'S3KeyGenerator' interface that you can implement and pass to the 'ExtendedClientConfiguration' to determine the generated s3 object's key.

Once I'm happy it works as required I'll do a pull request.

[Geeber]

+1 for this feature request; it would definitely come in handy for us, as it would make it easier (/possible) to set appropriate permissions when using one S3 bucket for multiple SQS queues.

[svoeller99]

#21 provides a very simple extension point that could be used to achieve this, as well.

[RajasGujarathi]

+1 for this request

[RajasGujarathi]

Was able to get a user-specified prefix working, but I think it is just for temporary proceeding ahead but the permanent clean fix will still be needed

<aws.java.sdk.version>2.17.100</aws.java.sdk.version>

pom.xml

<dependency>
      <groupId>software.amazon.awssdk</groupId>
      <artifactId>sqs</artifactId>
      <version>${aws.java.sdk.version}</version>
</dependency>
<dependency>
     <groupId>com.amazonaws</groupId>
     <artifactId>amazon-sqs-java-extended-client-lib</artifactId>
     <version>1.2.2</version>        
     <type>jar</type>
</dependency>

Java snippet

final AmazonS3 s3 = AmazonS3ClientBuilder.defaultClient();

StringBuilder s3KeyPrefix = new
    StringBuilder(s3BucketName).append(LocalDateTime.now());

ExtendedClientConfiguration extendedClientConfig = new ExtendedClientConfiguration()
                        .withPayloadSupportEnabled(s3, s3KeyPrefix.toString())
                        .withAlwaysThroughS3(Boolean.TRUE);

Note

• amazon-sqs-java-extended-client-lib uses version 1.x.x (Latest as of today is 1.2.2)
• If we make amazon-sqs-java-extended-client-lib to use version 2.x.x then the above implementation will not work and throw compilation exceptions and Runtime exceptions

[legosdenbbc]

I see there are now both
#21
and
#39

that attempt to address this issue, is it likely that either of these will get approved anytime soon?

EDIT: ahh i see these are now blocked due to being prior to e2ea202

[ayeletdasa]

+1 for this enhancement
we consider implementing the package ourselves from scratch as without this feature we can't use it 😕

[mike-harris0n]

This used to work in v1 of this library. Please fix this regression!

[ziyanli-amazon]

PR merged and new version released