better support for endpoint routing

[Downchuck]

Getting a lot of this:

gof3r error: 307: "Please re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests."

It'd be nice to have a little more debugging information. It seems that gof3r could handle the 307 as well.

It doesn't seem that the endpoint directive is really being picked up for use.

[rlmcpherson]

This error apparently occurs when the bucket being used has just been created and is also outside the S3 US standard region.

Unfortunately, the go standard library does not support 307 redirects automatically. (https://code.google.com/p/go/issues/detail?id=7912 )

It may be possible to handle this case in the s3gof3r transport implementation, but this will require some non-trivial changes to the client. In particular, retry logic may need to be moved into the transport with support to reset the body on retries.

[jfrost]

Interesting! I saw this a while ago and thought S3 was just having issues. Does the redirect stop after the bucket has been around for some time?

[rlmcpherson]

Yes, that's my understanding. This comment on this aws-cli issue has a good explanation: aws/aws-cli#634 (comment)

[Downchuck]

Outside of the standard 307 handling, is the "endpoint" directive from the .ini actually working? It seems that overriding it for put had no effect on the outcome.

[rlmcpherson]

I believe endpoint support is working correctly. Could you provide more information on the behavior you are seeing and include an example command?

[hydrajump]

@Downchuck I'm also having an issue overriding the default endpoint in the ini file.

This works,
gof3r cp ~/Desktop/test_file s3://hydrajump1234/test_file

This doesn't work,

$ cat gof3r.ini
[cp]
endpoint = s3.eu-central-1.amazonaws.com

gof3r cp ~/Desktop/test_file s3://hydrajump1234/test_file

gof3r error: 307: "Please re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests."

[rlmcpherson]

@hydrajump Is this an ini-specific problem? Does it work when specified as a command-line flag?

[rlmcpherson]

I successfully tested setting the endpoint for a new bucket in the Ireland region. There were no 307 responses, endpoint support is working correctly.

There is a separate problem with requests to the region you are trying to use above. Since the Frankfurt region supports only the new 'AWS Signature Version 4' signing method, that will need to be added for compatibility with this region. See #45

[hydrajump]

Thank you for investigating!

When you did you test with Ireland, did you modify the ini or specify it on the command line? If the latter can you please share your syntax, as I must be doing something wrong.

[rlmcpherson]

@hydrajump I've confirmed that setting the endpoint in the ini file does not work. This is because the endpoint also has a default value, which overrides the value from the ini file. I've opened an issue about this in jessevdk/go-flags#120

[trompx]

Hello @rlmcpherson, could you indicate the command you used to set the endpoint for Ireland eu-west-1 when doing a put or get ?
As discussed here, it's the time the dns propagates : https://forums.aws.amazon.com/message.jspa?messageID=196878

[rlmcpherson]

@trompx Use the --endpoint flag, as documented on the man page here: http://randallmcpherson.com/gof3r.html

For example: gof3r cp s3://<bucket>/<s3_path> <local_path> --endpoint=s3-eu-west-1.amazonaws.com