GetBucketLocation fails if region specified does not match the region the bucket is in.

[matthew-andrews]

There seems to be a regression of this issue: #380.

This is my code:-

awsSession := session.New()
s3Service := s3.New(awsSession, &aws.Config{
  LogLevel: aws.LogLevel(aws.LogDebugWithHTTPBody),
})

resp, err := s3Service.GetBucketLocation(&s3.GetBucketLocationInput{
  Bucket: aws.String(c.String("bucket")),
})
if err != nil {
  return cli.NewExitError(fmt.Sprintf("%s", err), 1)
}

When I set the AWS_REGION environment variable be different (us-east-1) to that of the bucket (eu-west-1), I get 403 errors:-

2016/06/11 15:06:31 DEBUG: Request s3/GetBucketLocation Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET http://s3.amazonaws.com/s3up-test?location= HTTP/1.1
Host: s3.amazonaws.com
User-Agent: aws-sdk-go/1.1.34 (go1.6; darwin; amd64)
[… AWS auth headers …]
Accept-Encoding: gzip


-----------------------------------------------------
2016/06/11 15:06:32 DEBUG: Response s3/GetBucketLocation Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Sat, 11 Jun 2016 14:06:31 GMT
Server: AmazonS3
X-Amz-Id-2: EGxuZn97P1HxAYJQLVZBqsyFM3zxgdJd2G7oIR1gGK9fZ1MH2r7D0fMx3wZo4IGdPR1TS83oAyM=
X-Amz-Request-Id: 54E6B7601DD1DC3B

f3
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>54E6B7601DD1DC3B</RequestId><HostId>EGxuZn97P1HxAYJQLVZBqsyFM3zxgdJd2G7oIR1gGK9fZ1MH2r7D0fMx3wZo4IGdPR1TS83oAyM=</HostId></Error>
0


-----------------------------------------------------
AccessDenied: Access Denied
    status code: 403, request id: 54E6B7601DD1DC3B

When the region is correct, the output is this:-

2016/06/11 15:07:28 DEBUG: Request s3/GetBucketLocation Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET http://s3-eu-west-1.amazonaws.com/s3up-test?location= HTTP/1.1
Host: s3-eu-west-1.amazonaws.com
User-Agent: aws-sdk-go/1.1.34 (go1.6; darwin; amd64)
[… AWS auth headers …]
Accept-Encoding: gzip


-----------------------------------------------------
2016/06/11 15:07:29 DEBUG: Response s3/GetBucketLocation Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Sat, 11 Jun 2016 14:07:29 GMT
Server: AmazonS3
X-Amz-Id-2: yz1z8vs+rFrZA+aGKSwSfN/moPNPrDLeYPU4Zh/amT8HoIqhYq9c3MkwL0ixs3aqWFdP/Vgk5KQ=
X-Amz-Request-Id: D949FAF64D0BD627

89
<?xml version="1.0" encoding="UTF-8"?>
<LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-03-01/">eu-west-1</LocationConstraint>
0


-----------------------------------------------------
{
  LocationConstraint: "eu-west-1"
}

I should be able to use the API/SDK to determine the location of an S3 bucket without knowing the location of the S3 bucket, shouldn't I?

[xibz]

Hello @matthew-andrews, thank you for bringing this to our attention. I cannot reproduce this. I've tried creating a bucket in eu-west-1 and still it succeeds. What Go and SDK versions are you using?

[jasdel]

@matthew-andrews I've been trying to reproduce this issue. I'm only able to encounter the AccessDesigned error when I don't have access to the bucket. For example I try to head your s3up-test bucket I get the error regardless of which region I make the request from. When I do a GetBucketLocation on a bucket I have permission to access I can call it the API successfully from any region. Do you have any policies setup on this bucket restricting the the region a request can be made from?

[matthew-andrews]

Hi, I've just tried again and can definitely reproduce the issue.

I've created a reduced test case here:-
https://github.com/matthew-andrews/s3-reduced

Which I have also reproduced on CircleCI (click make test to see the output):-

WITH EU REGION

export AWS_REGION=eu-west-1; echo $AWS_REGION && go run main.go;
eu-west-1
RESP:
{
  LocationConstraint: "eu-west-1"
}
ERR:
<nil>
----

WITH NO REGION

unset AWS_REGION; echo $AWS_REGION && go run main.go;

RESP:
{

}
ERR:
MissingRegion: could not find region configuration

The user is a plain empty AWS IAM user with no groups, no policies, etc. It is granted access to a bucket by a bucket policy attached to the bucket itself and that policy is this:-

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::862433486593:user/DeployUserFor_s3up"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::s3up-test/*",
                "arn:aws:s3:::s3up-test"
            ]
        }
    ]
}

I believe the go version on both my local system and in CircleCI is 1.6 and I think Circle probably pulls in this sdk from master. I just tested my local machine on v1.1.34.

Hope this helps!

[matthew-andrews]

Hm, actually that's a different error isn't it…

[matthew-andrews]

OK, try here — https://circleci.com/gh/matthew-andrews/s3-reduced/4

Click make test

EU REGION

export AWS_REGION=eu-west-1; echo $AWS_REGION && go run main.go;
eu-west-1
RESP:
{
  LocationConstraint: "eu-west-1"
}
ERR:
<nil>
----

WITH NO REGION

export AWS_REGION=us-east-1; echo $AWS_REGION && go run main.go;
us-east-1
RESP:
{

}
ERR:
AccessDenied: Access Denied
    status code: 403, request id: 740F596935D64CDE

[xibz]

Hello @matthew-andrews, I've reached out to s3 to further investigate this issue. Please let us know if there is anything else we can do for you.

[jasdel]

Hi @matthew-andrews are you still encountering this issue? It may be helpful to ask on the S3 AWS forums. Other users may of experienced the issue you are encountering, and will help get more visibility with your issue with S3.

[matthew-andrews]

I think it's a bug…?

[jasdel]

Hi @matthew-andrews I think this is a S3 service API issue, not SDK. This is why I was suggesting the S3 forums.

GetBucketLocation requires the requester to be the owner of the bucket.

The Writing IAM Policies: How to Grant Access to an Amazon S3 Bucket blog post goes through the process needed to grant permission to a bucket, including permission to get a bucket's location.

[matthew-andrews]

Ah you think the IAM permissions are wrong? I'll check that and get back to you. Apologies in advance if that's the case.

[jasdel]

I think that its a possibility, especially since you're able to get the bucket in one region but not another. I'm thinking that it is possible that there is a policy on the bucket that is limiting permissions to a specific region.

[jasdel]

Hi @matthew-andrews any luck identifying if this issue was caused by a policy rule?

[jasdel]

Hi @matthew-andrews We's escalated this issue to the S3 Service team, and they are currently investigating. We'll update once we have more information.

[jasdel]

Hi @matthew-andrews The S3 team got back with me and suggested the best API to use is HEAD bucket. GetBucketLocation uses a more complex permissions model where HeadBucket can be called by anyone. The bucket's region will be returned as a X-Amz-Bucket-Region header.

The HEAD bucket request can be made regardless of request signing, so anonymous (unsigned) requests can be made requesting a bucket's region.

I'm going to close this issue with this info. Please let us know if you have additional feedback, or issues.

[moofish32]

I'm not sure this issue should be closed. I can reproduce this still. If I have to set region to match the bucket location, what is the point of this API call at all? You can access the region from Config, and it must be set: https://docs.aws.amazon.com/sdk-for-go/api/aws/#Config. At this point the only useful purpose to the API call would be to check if the bucket is in the current region and iterate all regions creating a new Session for each region until you found the right one. Hopefully I'm missing something obvious, but for now I'm heading towards the HEAD workaround.

[owbmoore]

I'm not sure this issue should be closed. I can reproduce this still. If I have to set region to match the bucket location, what is the point of this API call at all? You can access the region from Config, and it must be set: https://docs.aws.amazon.com/sdk-for-go/api/aws/#Config. At this point the only useful purpose to the API call would be to check if the bucket is in the current region and iterate all regions creating a new Session for each region until you found the right one. Hopefully I'm missing something obvious, but for now I'm heading towards the HEAD workaround.

I agree, this should not have been closed. I have reproduced the problem just today when using this dependency:

	<dependency>
		<groupId>com.amazonaws</groupId>
		<artifactId>aws-java-sdk</artifactId>
		<version>1.11.747</version>
	</dependency>

Note: The suggested workaround above to use HeadBucket did not work for me using the Java API. It returned a 400 Bad Request when making the call on a bucket in a region different than the region used to create the AmazonS3 client.

A workaround which does work is to connect to each region in turn to determine which connection allows getBucketLocation to return without throwing an exception. The region which does not throw the exception is the region of the bucket. Pretty poor workaround but it does the job.

[metroidprototype]

Going to echo the previous sentiments and add some more. @jasdel can this be reopened?

I just did some more testing with both the HeadBucket path and GetBucketLocation path.

First, here is my code:

package main

import (
	"log"
	"net/http"
	"time"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/credentials"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
)

func main() {
	bucket := "168-dlx-one-staging-us-west-2-assets"
	region := "us-west-1"
	log.Printf("Testing %s with client in %s", bucket, region)
	headBucketRegion(bucket, region)
	getBucketLocation(bucket, region)

	region = "us-west-2"
	log.Printf("Testing %s with client in %s", bucket, region)
	headBucketRegion(bucket, region)
	getBucketLocation(bucket, region)

	region = "us-east-1"
	log.Printf("Testing %s with client in %s", bucket, region)
	headBucketRegion(bucket, region)
	getBucketLocation(bucket, region)

	region = "us-east-2"
	log.Printf("Testing %s with client in %s", bucket, region)
	headBucketRegion(bucket, region)
	getBucketLocation(bucket, region)
}

func getBucketLocation(bucket, region string) {
	c := getCli(region)

	input := &s3.GetBucketLocationInput{
		Bucket: aws.String(bucket),
	}

	result, err := c.GetBucketLocation(input)
	if err != nil {
		log.Printf("GetBucketLocation failed: %s", err.Error())
		return
	}

	log.Printf("GetBucketLocation %s location: %s",
		bucket,
		*result.LocationConstraint,
	)
}

func headBucketRegion(bucket, region string) {
	c := getCli(region)

	input := &s3.HeadBucketInput{
		Bucket: aws.String(bucket),
	}

	req, _ := c.HeadBucketRequest(input)

	err := req.Send()
	if err != nil {
		log.Printf("HeadBucket failed: %s", err.Error())
		return
	}

	log.Printf("HeadBucket %s region: %s",
		bucket,
		req.HTTPResponse.Header.Get("X-Amz-Bucket-Region"),
	)
}

func getCli(region string) *s3.S3 {
	cl := &http.Client{
		Timeout: 60 * 30 * time.Second,
	}

	creds := credentials.NewEnvCredentials()
	_, err := creds.Get()
	if err != nil {
		log.Printf("bad s3 credentials: %s", err)
		return nil
	}

	cfg := aws.NewConfig().
		WithRegion(region).
		WithCredentials(creds).
		WithMaxRetries(10).
		WithHTTPClient(cl)
	return s3.New(session.New(), cfg)
}

And here is the output:

2020/03/26 14:38:24 Testing 168-dlx-one-staging-us-west-2-assets with client in us-west-1
2020/03/26 14:38:25 HeadBucket failed: BucketRegionError: incorrect region, the bucket is not in 'us-west-1' region at endpoint ''
	status code: 301, request id: , host id:
2020/03/26 14:38:26 GetBucketLocation failed: AccessDenied: Access Denied
	status code: 403, request id: EB2818BE9D1502C8, host id: +KeyBs1bF9Q8jCZfAmpLeq4izmCdz/E94rYpHA4N+8KKEkKZtxrVtEAiiojwdWBuW6ZzGHQalrs=
2020/03/26 14:38:26 Testing 168-dlx-one-staging-us-west-2-assets with client in us-west-2
2020/03/26 14:38:26 HeadBucket 168-dlx-one-staging-us-west-2-assets region: us-west-2
2020/03/26 14:38:27 GetBucketLocation 168-dlx-one-staging-us-west-2-assets location: us-west-2
2020/03/26 14:38:27 Testing 168-dlx-one-staging-us-west-2-assets with client in us-east-1
2020/03/26 14:38:28 HeadBucket failed: BadRequest: Bad Request
	status code: 400, request id: 88F82FAA425B9252, host id: w82ARt4dbQ+UzKeCKyMh1q5Keg3sumXB8N+2Cm61y1RE+4/ESvCldlpINPLx/NiDkqVGbatdEMU=
2020/03/26 14:38:28 GetBucketLocation failed: AccessDenied: Access Denied
	status code: 403, request id: 3EAD2DC6D028FD58, host id: khsJrRtiYFm0g7CNKojSRRZiOatGrBwvTFoVIycVpsD4g3Ppg+ue6QYRAq6xO8w5D/7d734gl/s=
2020/03/26 14:38:28 Testing 168-dlx-one-staging-us-west-2-assets with client in us-east-2
2020/03/26 14:38:29 HeadBucket failed: BucketRegionError: incorrect region, the bucket is not in 'us-east-2' region at endpoint ''
	status code: 301, request id: , host id:
2020/03/26 14:38:30 GetBucketLocation failed: AccessDenied: Access Denied
	status code: 403, request id: 4D5B7B87060EFB8D, host id: eL4+X9uv2ZNALL0ZjnKLk26umRysZ4jhDrwpEEi9N2ZgYcIrEzcvZsv97f0PALkLjhQSWc24+7g=

You can see here using either method you must have your client region set to the region of the bucket for them to work. I feel like this is counterintuitive to a major (if not the primary) use case for GetBucketLocation: I have a bucket, but I do not know the region it exists in.

[metroidprototype]

After some discussion with AWS support I found out that the issue I was having with this behavior is the previously recommended solutions do NOT work when you are making cross account requests to HeadBucket and GetBucketLocation. A simple http HEAD does work and requires no credentials or cross account access in all cases for me. Here is my sample go code that I am now using:

func getBucketRegion(bucket string) (string, error) {
	url := fmt.Sprintf("https://%s.s3.amazonaws.com", bucket)
	res, err := http.Head(url)
	if err != nil {
		return "", err
	}
	return res.Header.Get("X-Amz-Bucket-Region"), nil
}

[anton-yurchenko]

@metroidprototype, this approach will throw an error in case your bucket "mutates" the endpoint URL.
For example, bucket www.website.com will cause an SSL error: Head "https://www.website.com.s3.amazonaws.com": x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not www.website.com.s3.amazonaws.com

---

The only working solution I found is to create a map of clients for each region, check where the bucket resides and use relevant client to perform operations on that bucket 🤯 🔫. Something like:

package main

import (
	"fmt"
	"os"
	"strings"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/endpoints"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
)

func main() {
	var cli map[string]*s3.S3 = make(map[string]*s3.S3)

	for _, partition := range endpoints.DefaultPartitions() {
		for region := range partition.Regions() {
			cli[region] = s3.New(session.Must(session.NewSession(&aws.Config{
				Region: aws.String(region),
			})))
		}
	}

	r, err := cli["us-east-1"].ListBuckets(&s3.ListBucketsInput{})
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}

	for _, bucket := range r.Buckets {
		l, err := cli["us-east-1"].GetBucketLocation(&s3.GetBucketLocationInput{
			Bucket: bucket.Name,
		})
		if err != nil {
			fmt.Println(err)
			os.Exit(1)
		}

		var region string
		if l.LocationConstraint == nil {
			region = "us-east-1"
		} else {
			region = *l.LocationConstraint
		}

		o, err := cli[region].GetBucketTagging(&s3.GetBucketTaggingInput{
			Bucket: bucket.Name,
		})
		if err != nil && !strings.Contains(err.Error(), "The TagSet does not exist") {
			fmt.Println(err)
			os.Exit(1)
		} else if err == nil {
			fmt.Println(o.TagSet)
		}
	}
}

I believe the solution should be cross region calls (blog):

New pseudo region: aws-global to make cross region requests

But I was not able to operate against aws-global region 😢