Release 4.2.0 changes authentication behaviour

[morfien101]

There appears to be a change in behaviour with the AWS Creds chain.

On version 3.74.0 the ENV vars are selected first and requests will us these values for the API calls.
In version 4.2.0 the EC2 Instance role appears to be selected first, in our case this causes authentication failures.

We found this with our Jenkins agents. We pass in credentials to the agent using the withCredentials function which will set the auth env vars AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN. However we get a failure like this:

Error: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 648d56a7-dc69-4126-9bae-80689030cc83, api error AccessDenied: User: arn:aws:sts::123:assumed-role/jenkins-agents-ec2/i-123abc is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::123:role/jenkins-role

This is not the case in 3.74.0

Maybe this change?
#23282

[gdavison]

Hi @morfien101, thanks for opening this issue. Version 4 of the AWS provider has changes to the authentication flow to bring it in line with the behaviour of the AWS CLI and the default behaviour of the AWS SDKs. For more information, see https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-4-upgrade#changes-to-authentication.

Most cases where we've seen this reported define both a profile in the provider configuration block and the credentials environment variables, i.e.

provider "aws" {
  profile = "example-profile"
}

If that is not the case, can you please share the details of your provider configuration block?

[morfien101]

We do define the profile in the provider block as you stated.

Our use case is like so:
Dev on laptop has .aws/[config,credentails] with the profiles. This works fine.
On Jenkins server, the EC2 instance has an instance profile assigned to it that has basic abilities in it to allow the starting of the agents. Then the Jenkins config has credentials that are passed into the job which appear as the environment variables.
As stated in the original post, the env vars are ignored due to the presence of the Instance Profile.

[gdavison]

Thank you for your response, @morfien101. This is expected behaviour starting with version 4.

Version 4 of the AWS provider has changes to the authentication flow to bring it in line with the behaviour of the AWS CLI and the default behaviour of the AWS SDKs. For more information, see https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-4-upgrade#changes-to-authentication.

[EdSforzati]

Hi @gdavison, since this is expected now, and the gospel spread far and wide in the past was "use the profile parameter in the provider blocks", is there a recommended new way to use this provider? I'm in the same boat as @morfien101 above.

I'm expecting a lot of people to be hit by this problem in the future, and think an official recommendation would perhaps be better than just shutting the door without any further discourse.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.