Use us-gov-west-1 for global APIs in aws-us-gov #9947
Conversation
Certain partition-global AWS services, like IAM, seem to require specific regions. In the regular 'aws' partition, this is us-east-1. In the 'aws-us-gov' partition, this is us-gov-west-1. Providing us-gov-east-1 returns an error from AWS: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'us-gov-east-1'. This resolves a problem where AWS authentication could randomly fail depending on the value cached by Vault at startup.
|
|
|
@bluekeyes Thanks for this! Do you have a link to AWS docs which references the specialness of "us-gov-west-1" that you could add to the code comment? To confirm, were you able to test this out in the us gov partition? Running such a test is a little more challenging for us so 💯 if you've verified it. |
|
I wish I could find this fact in the docs, but I've spent a while looking and can't find this called out in either the IAM docs or the region and endpoint docs. The closest reference I have is that both For testing, I haven't tried deploying a binary with this change to our GovCloud environment yet (although I have reproduced the original issue multiple times.) I can test this out and report back if it will help. |
|
I deployed a custom Vault binary with this patch in our staging GovCloud environment and it solves the problem. I was able to step down Vault 15 times while simultaneously authenticating from another host without any failures. For comparison, when I first connected to the environment, authentication was broken and I had to step down Vault four times before it cached the correct region and fixed authentication. |
|
@bluekeyes Reading the original issue prompted an additional test. Is it possible for you to run this within |
|
@kalafut I launched a Vault server built from my fork in us-gov-east-1, configured the AWS auth backend in a similar way to what we use in us-gov-west-1, and was able to authenticate with IAM from an instance in us-gov-east-1, so I believe this will work fine in both regions. |
|
@bluekeyes Thanks for the extra test! |
| // because it is always enabled (and enabled for STS) by default. | ||
| // However, there are a few exceptions: | ||
| // | ||
| // For "aws", choose "us-east-1" because it is always enabled (and |
There was a problem hiding this comment.
Really appreciate the extended comments in here!
* Use us-gov-west-1 for global APIs in aws-us-gov Certain partition-global AWS services, like IAM, seem to require specific regions. In the regular 'aws' partition, this is us-east-1. In the 'aws-us-gov' partition, this is us-gov-west-1. Providing us-gov-east-1 returns an error from AWS: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'us-gov-east-1'. This resolves a problem where AWS authentication could randomly fail depending on the value cached by Vault at startup.
* Use us-gov-west-1 for global APIs in aws-us-gov Certain partition-global AWS services, like IAM, seem to require specific regions. In the regular 'aws' partition, this is us-east-1. In the 'aws-us-gov' partition, this is us-gov-west-1. Providing us-gov-east-1 returns an error from AWS: SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'us-gov-east-1'. This resolves a problem where AWS authentication could randomly fail depending on the value cached by Vault at startup. Co-authored-by: Billy Keyes <[email protected]>
|
Is this still an issue for us-iso and us-isob? It appears that this fix doesn't resolve this issue for other non-commercial partitions. |
Certain partition-global AWS services, like IAM, seem to require specific regions. In the regular 'aws' partition, this is us-east-1. In the 'aws-us-gov' partition, this is us-gov-west-1. Providing us-gov-east-1 returns an error from AWS:
This resolves a problem where AWS authentication could randomly fail depending on the value cached by Vault at startup.
Fixes #9935.