S3 sign-uri needs a uri-encode?

[mflatt]

I think that line 634 of "s3.rkt" needs a uri-encode around bucket+path as the argument to canonical-string-to-sign.

I ran into problems trying upload with path elements that contain "@", and adding uri-encode there solved the problem.

[greghendershott]

The following interaction worked successfully for me:

> (require aws)
> (put/bytes "greghendershott.com/[email protected]" #"Hi there" "text")
"HTTP/1.1 200 OK\r\nx-amz-id-2: NLQkvIXPLPCPnpH5H/bKwaI3jIVFPmsZLfslBIb/7t2BGE/Z5XCpF3bLCs72ZlJJ\r\nx-amz-request-id: 052B9FFB885A2A75\r\nDate: Wed, 16 Jul 2014 14:52:13 GMT\r\nETag: \"d9385462d3deff78c352ebb3f941ce12\"\r\nContent-Length: 0\r\nServer: AmazonS3\r\n\r\n"
> (get/bytes "greghendershott.com/[email protected]")
#"Hi there"
> (sign-uri "greghendershott.com/[email protected]" "GET" (+ (current-seconds) 3600) '())
"http://greghendershott.com.s3.amazonaws.com/[email protected]?AWSAccessKeyId=1GW9JZAP3A5DMMM7FF82&Expires=1405526079&Signature=xiuMQH52tGNATIEdw9%2FWt4%2BRSU8%3D"
> (require net/url)
> (define in (get-impure-port (string->url "http://greghendershott.com.s3.amazonaws.com/[email protected]?AWSAccessKeyId=1GW9JZAP3A5DMMM7FF82&Expires=1405526079&Signature=xiuMQH52tGNATIEdw9%2FWt4%2BRSU8%3D")))
> (port->string in)
"HTTP/1.1 200 OK\r\nx-amz-id-2: Xao51pCy7hsWojJjA8RKoxoKoXHKfDC8nLN/uGX71rpXp6lE2Xfq246lVn6+YCjV\r\nx-amz-request-id: 5E01037C71BD3E42\r\nDate: Wed, 16 Jul 2014 14:55:30 GMT\r\nLast-Modified: Wed, 16 Jul 2014 14:52:13 GMT\r\nETag: \"d9385462d3deff78c352ebb3f941ce12\"\r\nAccept-Ranges: bytes\r\nContent-Type: text\r\nContent-Length: 8\r\nServer: AmazonS3\r\n\r\nHi there"
> (close-input-port in)

So I can't seem to reproduce this.

I think that line 634 of "s3.rkt" needs a uri-encode around bucket+path as the argument to canonical-string-to-sign.

Line 634 is in the sign-uri function. Is that what you meant -- getting a temporarily valid URI to an already-uploaded object? Because you mentioned problems with uploading. Anyway, sign-uri worked OK for me with the object named [email protected], in the interactions above.

I'm leery of just "defensively" adding the uri-encode where you suggest, because S3 authentication is sensitive. I don't want to create a problem where it fails because URI for the auth encryption is uri-encoded, but not the plain URI in the request.

However if you can help me figure out the problem you're experiencing, of course I'll dig in and figure it out!

[mflatt]

Sorry for the poor report. For some reason, I can't replicate the problem at the moment, either. I'll investigate further.

[mflatt]

It appears that I was giving put/bytes an already-encoded path. Then, for some reason, I can only get it to fail at an Oregon bucket (not Standard or EU).

Here's the example that produces an error:

 (require aws)
 (put/bytes "test.racket-lang.org-oregon/a%40b" #"Hi there" "text") 

I think I encoded the path given to put/bytes because I was running into trouble without it, as illustrated by

(put/bytes "test.racket-lang.org-oregon/a%b" #"Hi there" "text")

which gives me an "Invalid URI" error.

[greghendershott]

Thank you for the additional information!

I'm going to need to dig into this when I have a little more time -- to understand why the bucket regions are behaving differently and what it means. And to feel like I have a crisp understanding when/where to uri-encode or not. Will try to follow-up within 1-2 days.

[greghendershott]

OK I looked at this again. I'm still not able to reproduce, even with an Oregon bucket. Interactions:

s3.rkt> (create-bucket "test-mflatt-issue" "us-west-2")
"HTTP/1.1 200 OK\r\nx-amz-id-2: GeY8kJ97BKsiQfhqacHfCi3cpyH53uQZ1t4oUjaERjfjGXTNkkY5sgwUCao+Ennb\r\nx-amz-request-id: 1043B1EA921F9617\r\nDate: Fri, 18 Jul 2014 20:52:26 GMT\r\nLocation: http://test-mflatt-issue.s3.amazonaws.com/\r\nContent-Length: 0\r\nServer: AmazonS3\r\n\r\n"

s3.rkt> (put/bytes "test-mflatt-issue/a@b" #"Hi there" "text")
"HTTP/1.1 200 OK\r\nx-amz-id-2: gesr/pR2k2nFifxMdW7KjQ02iaRKqomPj9/GWWpyw7sy+H1oINxA1kCSJpLeIKFYiYPIgFst6S0=\r\nx-amz-request-id: 7D379F235EF55EB2\r\nDate: Fri, 18 Jul 2014 20:59:25 GMT\r\nETag: \"d9385462d3deff78c352ebb3f941ce12\"\r\nContent-Length: 0\r\nServer: AmazonS3\r\n\r\n"

s3.rkt> (get/bytes "test-mflatt-issue/a@b")
#"Hi there"

s3.rkt> (sign-uri "test-mflatt-issue/a@b" "GET" (+ 3600 (current-seconds)) '())
"http://test-mflatt-issue.s3.amazonaws.com/a@b?AWSAccessKeyId=1GW9JZAP3A5DMMM7FF82&Expires=1405720852&Signature=4pHd8QhfQBzJPWwWgPpeKFS4i2A%3D"

And that URI http://test-mflatt-issue.s3.amazonaws.com/a@b?AWSAccessKeyId=1GW9JZAP3A5DMMM7FF82&Expires=1405720852&Signature=4pHd8QhfQBzJPWwWgPpeKFS4i2A%3D in a browser gives me "Hi there" text response.

(Note: That URI may have expired by the time you read this.)

So:

Using @ in the S3 object pathname doesn't cause an issue for me, with Oregon region, when putting the object, getting it, or making a pre-signed URI for it.

I wonder what we're doing differently?

[mflatt]

Well, this is frustrating.

The examples that failed for me previously were

(put/bytes "test.racket-lang.org-oregon/a%40b" #"Hi there" "text") 

and

(put/bytes "test.racket-lang.org-oregon/a%b" #"Hi there" "text")

That is, it failed if I encode in the case of "@", and it failed if I didn't encode (for understandable reasons) in the case of "%". Things worked fine for me if I refrained from encoding "@", as in your examples.

BUT, it's working for me now. It's possible that I was somehow consistently confused, but I struggled for quite a while to get pkg-build.racket-lang.org synced the first time, I was trying from multiple networks (Utah, MSR Cambridge, and apartment Cambridge), and I derived the example above a full day later. It works for me now from multiple networks, so maybe it doesn't matter whether AWS changed or how I managed to be confused.

[mflatt]

I'm able to replicate the original problem by creating a new bucket. Maybe the issue is not related to "us-west-2" but the newness of the bucket, since I had tried US Standard and EU using existing test buckets.

Just to be clear, the key part of the example is that using "%40" instead of "@" in the object name.

laptop% racket
Welcome to Racket v6.1.0.3.
Note: readline loaded
> (require aws)
> (create-bucket "test3.racket-lang.org" "us-west-2")
"HTTP/1.1 200 OK\r\nx-amz-id-2: GXmV9RQaWL3WJf+LiLXayNISbTk7hfC4Gxut981iHCbD3/adrAXK4J7Koz4pkSzv\r\nx-amz-request-id: 6D2C3E42FCB70EC4\r\nDate: Sun, 20 Jul 2014 06:05:09 GMT\r\nLocation: http://test3.racket-lang.org.s3.amazonaws.com/\r\nContent-Length: 0\r\nServer: AmazonS3\r\n\r\n"
> (put/bytes "test3.racket-lang.org/a%40c" #"Hi there" "text")
aws: HTTP/1.1 403 Forbidden
x-amz-request-id: A426E7182D3AFD98
x-amz-id-2: syXAx2WFRS41UeHGe04+/YOGVKZHGgosD7SJf35Vs36TKHOFMCPt8BQICSrEEWFs
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Sun, 20 Jul 2014 06:05:17 GMT
Connection: close
Server: AmazonS3

 <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><StringToSignBytes>50 55 54 0a 32 54 68 55 59 74 50 65 2f 33 6a 44 55 75 75 7a 2b 55 48 4f 45 67 3d 3d 0a 74 65 78 74 0a 53 75 6e 2c 20 32 30 20 4a 75 6c 20 32 30 31 34 20 30 36 3a 30 35 3a 31 36 20 47 4d 54 0a 2f 74 65 73 74 33 2e 72 61 63 6b 65 74 2d 6c 61 6e 67 2e 6f 72 67 2f 61 40 63</StringToSignBytes><RequestId>A426E7182D3AFD98</RequestId><HostId>syXAx2WFRS41UeHGe04+/YOGVKZHGgosD7SJf35Vs36TKHOFMCPt8BQICSrEEWFs</HostId><SignatureProvided>f1HBRmTknYI4IDIVQ5M68p70N5A=</SignatureProvided><StringToSign>PUT
2ThUYtPe/3jDUuuz+UHOEg==
text
Sun, 20 Jul 2014 06:05:16 GMT
/test3.racket-lang.org/a@c</StringToSign><AWSAccessKeyId>AKIAJ2NAR7EK3W2AWLYQ</AWSAccessKeyId></Error>
HTTP 403 "Forbidden". AWS Code="SignatureDoesNotMatch" Message="The request signature we calculated does not match the signature you provided. Check your key and signing method."
  context...:
   check-response
   request/redirect/uri
   /Users/mflatt/plt/racket/collects/racket/private/misc.rkt:87:7