Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

s3 bucket existence - unexpected behavior #796

Closed
s2t2 opened this issue Apr 28, 2015 · 2 comments
Closed

s3 bucket existence - unexpected behavior #796

s2t2 opened this issue Apr 28, 2015 · 2 comments

Comments

@s2t2
Copy link

s2t2 commented Apr 28, 2015

The code below demonstrates unexpected behavior when working with buckets in version 2.0.4.1. See also #460 (comment).

creds = Aws::Credentials.new(KEY_ID, KEY_SECRET)

# A) CLIENT
client = Aws::S3::Client.new(:region => REGION, :credentials => creds)
bucket1 = Aws::S3::Bucket.new(:name => BUCKET_NAME, :client => client)
bucket1.exists? #> Aws::S3::Errors::Http301Error: from /usr/local/var/rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/aws-sdk-resources-2.0.41/lib/aws-sdk-resources/resource.rb:134:in `rescue in exists?'
bucket1.create(:acl => "private") #> Aws::S3::Errors::BucketAlreadyExists: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.

# B) RESOURCE
resource = Aws::S3::Resource.new(:region => REGION, credentials: creds)
bucket2 = resource.create_bucket(:bucket => BUCKET_NAME, :acl => "private")
bucket2.exists? #> true
@trevorrowe
Copy link
Member

Thank you for reporting the issue. The #exists? check works by making a HEAD request to the bucket. Because we do not know the actual region of the bucket, that may not exist at all, the HEAD request is being sent to the currently configured region for the Aws::S3::Client.

It looks like Amazon S3 returns 301 permanent redirects for existing buckets when you head it in the wrong region. The fix would be to treat this status code as a successful check. Yes the bucket exists.

@trevorrowe
Copy link
Member

The fix was very straight forward. I've added a test to cover regressions. This should go out later this week. Thank you for submitting the bug report and for your patience!

@srchase srchase added the bug label Dec 24, 2018
czimergebot pushed a commit to chanzuckerberg/czid-web that referenced this issue Oct 30, 2019
[IDSEQ-1643] Fix bug for cross-region S3 uploads### Description
- This is just to address the cross-region copying issue from last week. See JIRA for details.
- Using the S3 "global" non-regional specific endpoint means that AWS will resolve the region itself.

### Notes
- This was the most elegant solution I could find. My original proposal was to use `GetBucketLocation` but apparently it is only for the bucket owner (https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLocation.html). I also found suggestions to use head bucket but got 301 redirects / couldn't figure out how to get the region info from that (aws/aws-sdk-ruby#796).
- This was the only use of S3_CLIENT that I found necessary to change, and it was the only one changed in the previous PR (#2538).
- Why not set this global endpoint all the time? AWS recommends against it "Please configure the proper region to avoid multiple unnecessary redirects and signing attempts". And for most other requests we're in our own buckets in our known region.

### Tests
- Testing with the reported user URL.

#### Before:
![Screen Shot 2019-10-29 at 2 59 37 PM](https://user-images.githubusercontent.com/5652739/67895204-4a3f6900-fb17-11e9-8a60-c4b9b05d36b6.png)

#### After:
![Screen Shot 2019-10-29 at 3 02 12 PM](https://user-images.githubusercontent.com/5652739/67895214-4f9cb380-fb17-11e9-81b9-698bc63ff13d.png)

#### Ran samples end-to-end locally and it works: 
<img width="1088" alt="Screen Shot 2019-10-30 at 12 53 59 PM" src="https://user-images.githubusercontent.com/5652739/67895228-54f9fe00-fb17-11e9-9d5b-69c19cbcf898.png">
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants