DefaultCredentialsProvider is not working for ECS instances

[MatteCarra]

I was testing a simple program which invokes a lambda.
The program is running nomrmally on my pc with aws credentials setup with aws configure,
but when I deploy it to ecs with the right role it doesn't work and this exception is thrown: https://pastebin.com/P23q2By1.

My code:

val region = "eu-west-1"
val credentialsV2 = DefaultCredentialsProvider.builder().build()
val lambda: LambdaAsyncClient = LambdaAsyncClient.builder().credentialsProvider(credentialsV2).region(Region.of(region)).build()
lambda.invoke(
  InvokeRequest.builder()
    .functionName(myLambda)
      .payload(ByteBuffer.wrap(myBody.toString().getBytes("UTF-8")))
    .build()
).toScala

[jonmcewen]

I see the same with DynamoDB. Credentials load from file, but not from EC2 instance profile

[eliaslevy]

Ditt for S3. Confirmed the instance profile works just fine by using the AWS CLI tools.

[frankbregulla1111]

I think the problem is that the ElasticContainerCredentialsProvider checks whether environment variable AWS_CONTAINER_CREDENTIALS_PATH is present whereas it is called AWS_CONTAINER_CREDENTIALS_RELATIVE_URI in the container on ECS.
(ElasticContainerCredentialsProvider.java:89)

[talton-rmn]

Edit: Looks like my comment is related to issue #120

I'm running into a similar problem. I added some debug code that would try to get the credentials from each provider individually, and compare them to the results of the DefaultCredentialProvider.

17:54:47 a.ELBWatcher - Env Creds: Failure(software.amazon.awssdk.SdkClientException: Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).)

17:54:47 a.ELBWatcher - System Creds: Failure(software.amazon.awssdk.SdkClientException: Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).)

17:54:47 a.ELBWatcher - Profile Creds: Success(null)

17:54:47 a.ELBWatcher - Container Creds: Failure(software.amazon.awssdk.SdkClientException: Credentials cannot be loaded from ECS because the ECS credentials environment variable (AWS_CONTAINER_SERVICE_ENDPOINT) and system property (aws.containerServiceEndpoint) are not set or cannot be accessed due to the security manager.)

17:54:47 a.ELBWatcher - Instance Creds: Success(AwsSessionCredentials(<scrubbed>))

17:54:47 a.ELBWatcher - DefaultCredProvider creds: null

It looks like the problem is that software.amazon.awssdk.auth.ProfileCredentialsProvider is returning null instead of throwing an error. AwsCredentialsProviderChain relies on the provider throwing an error, and doesn't check for null values:

    for (AwsCredentialsProvider provider : credentialsProviders) {
        try {
            AwsCredentials credentials = provider.getCredentials();

            log.debug("Loading credentials from {}", provider.toString());

            lastUsedProvider = provider;
            return credentials;
        } catch (RuntimeException e) {
            // Ignore any exceptions and move onto the next provider
            log.debug("Unable to load credentials from {}:{}", provider.toString(), e.getMessage(), e);
        }
    }

[eliaslevy]

Might be fixed in master by 62f2d8e. Did you try master?

[talton-rmn]

It looks like -preview-2 is still the latest version available from maven ( https://mvnrepository.com/artifact/software.amazon.awssdk/ec2 ).

Any suggestions on how to use the master branch as a dependency in another project?

[eliaslevy]

You'd have to build a jar from master yourself and drop it on your project's unmanaged dependencies directory.

[MatteCarra]

@eliaslevy I don't think so, but I think @ffriedrich is right.