Aws::S3::Client.stub_responses doesn't stub metadata anymore?

[porbas]

I'm in course of upgrading aws-sdk gems in my projects. Unfortunately my tests goes red 😞

In aws-sdk v 2.2.37 I could stub metadata:

piotrek@t530:~$ aws.rb _2.2.37_
Aws> s3 = Aws::S3::Client.new(stub_responses: true)
=> #<Aws::S3::Client>
Aws> s3.stub_responses :head_object, metadata: {'custom_key' => 'abc'}
=> [{:http=>
   #<Seahorse::Client::Http::Response:0x00000006269700
    @body=#<StringIO:0x00000006248398>,
    @complete=false,
    @done=nil,
    @error=nil,
    @headers={"x-amzn-requestid"=>"stubbed-request-id", "x-amz-meta-custom_key"=>"abc"},
    @listeners={},
    @status_code=200>}]
Aws> s3.head_object bucket: 'bucket', key: 'key'
[Aws::S3::Client 200 0.242652 0 retries] head_object(bucket:"bucket",key:"key")  
=> #<struct Aws::S3::Types::HeadObjectOutput
 delete_marker=nil,
 accept_ranges=nil,
 expiration=nil,
 restore=nil,
 last_modified=nil,
 content_length=nil,
 etag=nil,
 missing_meta=nil,
 version_id=nil,
 cache_control=nil,
 content_disposition=nil,
 content_encoding=nil,
 content_language=nil,
 content_type=nil,
 expires=nil,
 website_redirect_location=nil,
 server_side_encryption=nil,
 metadata={"custom_key"=>"abc"},
 sse_customer_algorithm=nil,
 sse_customer_key_md5=nil,
 ssekms_key_id=nil,
 storage_class=nil,
 request_charged=nil,
 replication_status=nil>

In current 2.6.28:

piotrek@t530:~$ aws.rb _2.6.28_
Aws> s3 = Aws::S3::Client.new(stub_responses: true)
=> #<Aws::S3::Client>
Aws> s3.stub_responses :head_object, metadata: {'custom_key' => 'abc'}
=> [{:http=>
   #<Seahorse::Client::Http::Response:0x00000006069ce8
    @body=#<StringIO:0x000000060465e0>,
    @complete=false,
    @done=nil,
    @error=nil,
    @headers={"x-amzn-requestid"=>"stubbed-request-id", "x-amz-meta-custom_key"=>"abc"},
    @listeners={},
    @status_code=200>}]
Aws> s3.head_object bucket: 'bucket', key: 'key'
[Aws::S3::Client 200 0.245525 0 retries] head_object(bucket:"bucket",key:"key")  
=> #<struct Aws::S3::Types::HeadObjectOutput
 delete_marker=nil,
 accept_ranges=nil,
 expiration=nil,
 restore=nil,
 last_modified=nil,
 content_length=nil,
 etag=nil,
 missing_meta=nil,
 version_id=nil,
 cache_control=nil,
 content_disposition=nil,
 content_encoding=nil,
 content_language=nil,
 content_type=nil,
 expires=nil,
 expires_string=nil,
 website_redirect_location=nil,
 server_side_encryption=nil,
 metadata={},
 sse_customer_algorithm=nil,
 sse_customer_key_md5=nil,
 ssekms_key_id=nil,
 storage_class=nil,
 request_charged=nil,
 replication_status=nil,
 parts_count=nil>

Is it error or should I stub metadata other way?

[awood45]

I'm able to recreate this, taking a look.

[awood45]

It looks like this changed with PR #1216

[awood45]

What's interesting is that the PR above seems to create the correct HTTP response, with metadata in the header instead of in the response body (which obviously would not exist with a #head_object call). I'm going to check if there's a deeper issue here.

[awood45]

Self Note: This doesn't appear to be broken for actual calls to the S3 endpoint (thankfully), checking for differences in stubbed and end-to-end responses.

[awood45]

The responses seem to be the same at most levels, the issues occurs somewhere deep in the XML parsing stack.

[awood45]

It appears that body parsing of the stubbed XML response for #head_object causes the issue here. It's clobbering the metadata value exclusively when body parsing happens.

[awood45]

Additionally, this issue only occurs for #head_object, and not for #get_object.

[awood45]

I've cut a PR that fixes the symptoms of this issue, but I'm going to have it reviewed before shipping, as there are a couple open problems. In any case, thanks for raising this.

[awood45]

We've pushed a fix for this that will go out with the next release.