put object body type

[fengjiansunren]

Describe the bug

c.Request.Body is received from http request, then sdk-v2 client use this body

uploadInput := &awsS3.PutObjectInput{
		Bucket: &bucketName,
		Key:    &objectName,
		Body:   c.Request.Body,
}

func (s *Client) PutObject(ctx context.Context, input *s3.PutObjectInput) (*s3.PutObjectOutput, error) {
	result, err := s.client.PutObject(ctx, input)
	if err != nil {
		xlog.Error("Failed to put object", xlog.String("bucket", *(input.Bucket)),
			xlog.String("object", *(input.Key)), xlog.String("error", err.Error()))
		return nil, err
	}
	return result, nil
}

Expected Behavior

no error

Current Behavior

err message

failed to rewind transport stream for retry, request stream is not seekable
request stream is not seekable

Reproduction Steps

sdk-v2 client putobject use body from http request

Possible Solution

No response

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

	github.com/aws/aws-sdk-go v1.44.200
	github.com/aws/aws-sdk-go-v2 v1.17.4
	github.com/aws/aws-sdk-go-v2/config v1.18.13
	github.com/aws/aws-sdk-go-v2/credentials v1.13.13
	github.com/aws/aws-sdk-go-v2/service/s3 v1.30.3
	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.10 // indirect
	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.22 // indirect
	github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.28 // indirect
	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.22 // indirect
	github.com/aws/aws-sdk-go-v2/internal/ini v1.3.29 // indirect
	github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.20 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.11 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.23 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.22 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.13.22 // indirect
	github.com/aws/aws-sdk-go-v2/service/sso v1.12.2 // indirect
	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.2 // indirect
	github.com/aws/aws-sdk-go-v2/service/sts v1.18.3 // indirect

Compiler and Version used

1.19.2

Operating System and version

centos 7.6

[Thomasdezeeuw]

I've hit this issue as well @fengjiansunren , see #1108.

For now I'm working around it by disabling retrying (returning the original error):

noRetry := func(options *s3.Options) {
	options.RetryMaxAttempts = 1
}
output, err := w.client.PutObject(ctx, input, noRetry)

[RanVaknin]

Hi @fengjiansunren ,

Is your body an io.ReadSeeker? The SDK needs to seek through it to calculate the content-length on putObject request. My guess is this is why you are seeing this error.

The solution provided on #1108 is correct. You can also look at #1938 to see if the solution there is applicable.

Let us know if you have any other questions.
Thanks,
Ran~

[fengjiansunren]

@Thomasdezeeuw
thanks for your solution, now it's work like this

client := s3.NewFromConfig(cfg, func(o *s3.Options) {
			o.RetryMaxAttempts = 1
})

result, err := client.PutObject(ctx, input, s3.WithAPIOptions(
		v4.SwapComputePayloadSHA256ForUnsignedPayloadMiddleware,)

[fengjiansunren]

Hi @fengjiansunren ,

Is your body an io.ReadSeeker? The SDK needs to seek through it to calculate the content-length on putObject request. My guess is this is why you are seeing this error.

The solution provided on #1108 is correct. You can also look at #1938 to see if the solution there is applicable.

Let us know if you have any other questions. Thanks, Ran~

@RanVaknin
no, my body is an io.ReadCloser, it's received from another http request, i want use this request.body as s3 client putObjectInput.Body, this will not copy the memory

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[wguo-github]

Hi @RanVaknin, just wanted to mention that we're seeing a similar issue (as of v1.31.0 of the S3 SDK) where a retry fails due to not being able to rewind the stream. My assumption is that (as mentioned above), when it comes to retry handling the SDK actually requires an io.ReadSeeker in order to be able to rewind the input stream. Whilst that's fine, the question would be whether this should be reflected in the PutObjectInput declaration, which still just requires a plain io.Reader? (ref) Is this an inconsistency in the SDK, or is there some other reason it's been done this way? Thanks!

[tetra12]

The issue still persists for Minio and newer SDK:

github.com/aws/aws-sdk-go-v2 v1.21.0
github.com/aws/aws-sdk-go-v2/config v1.18.42
github.com/aws/aws-sdk-go-v2/service/s3 v1.39.0

Resolved with a workaround from @fengjiansunren

[fengjiansunren]

您好！已收到！