Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed the issue where CRT-based S3 client was using excessive memory #3800

Merged
merged 5 commits into from
Mar 31, 2023
+19 −2
Merged

Fixed the issue where CRT-based S3 client was using excessive memory #3800

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c4fdee9
Fixed the issue where AWS CRT-based S3 client was eagerly buffering d…
zoewangg Feb 28, 2023
242a59c
Merge branch 'master' into zoewang/tm-memoryIssue
zoewangg Mar 7, 2023
2c10396
Merge branch 'master' into zoewang/tm-memoryIssue
zoewangg Mar 22, 2023
0582074
Merge branch 'master' into zoewang/tm-memoryIssue
zoewangg Mar 28, 2023
29f72eb
Merge branch 'master' into zoewang/tm-memoryIssue
zoewangg Mar 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changes/next-release/bugfix-AWSCRTbasedS3client-7316932.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"type": "bugfix",
"category": "AWS CRT-based S3 client",
"contributor": "",
"description": "Fixed the issue where AWS CRT-based S3 client was eagerly buffering data before the underlying CRT component was able to handle it. See [#3726](https://github.com/aws/aws-sdk-java-v2/issues/3726)"
}
6 changes: 6 additions & 0 deletions .changes/next-release/bugfix-AWSCRTbasedS3client-89a9c8e.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"type": "bugfix",
"category": "AWS CRT-based S3 client",
"contributor": "",
"description": "Reduced the buffer used to upload object from 16MB to 1MB."
}
9 changes: 7 additions & 2 deletions ...n/java/software/amazon/awssdk/services/s3/internal/crt/S3CrtRequestBodyStreamAdapter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
package software.amazon.awssdk.services.s3.internal.crt;

import java.nio.ByteBuffer;
import java.util.concurrent.atomic.AtomicBoolean;
import software.amazon.awssdk.annotations.SdkInternalApi;
import software.amazon.awssdk.crt.http.HttpRequestBodyStream;
import software.amazon.awssdk.http.async.SdkHttpContentPublisher;
Expand All @@ -26,19 +27,23 @@
*/
@SdkInternalApi
public final class S3CrtRequestBodyStreamAdapter implements HttpRequestBodyStream {
private static final long MINIMUM_BYTES_BUFFERED = 16 * 1024 * 1024L;
private static final long MINIMUM_BYTES_BUFFERED = 1024 * 1024L;
private final SdkHttpContentPublisher bodyPublisher;
private final ByteBufferStoringSubscriber requestBodySubscriber;

private final AtomicBoolean subscribed = new AtomicBoolean(false);

public S3CrtRequestBodyStreamAdapter(SdkHttpContentPublisher bodyPublisher) {
this.bodyPublisher = bodyPublisher;
this.requestBodySubscriber = new ByteBufferStoringSubscriber(MINIMUM_BYTES_BUFFERED);
bodyPublisher.subscribe(requestBodySubscriber);
}

@Override
public boolean sendRequestBody(ByteBuffer outBuffer) {
if (subscribed.compareAndSet(false, true)) {
bodyPublisher.subscribe(requestBodySubscriber);
}

// blocking here because CRT S3 requires the buffer to be completely filled
return requestBodySubscriber.blockingTransferTo(outBuffer) == ByteBufferStoringSubscriber.TransferResult.END_OF_STREAM;
}
Expand Down