Aws::Errors::MissingCredentialsError: unable to sign request without credentials set

[rayway30419]

Hi

I use IAM-role to assign policy to my EC2 instance, but sometimes it raise missing credential error.

Aws::Errors::MissingCredentialsError: unable to sign request without credentials set

Does anyone have same problem?

[awood45]

So you're saying that on an EC2 instance, you're seeing this periodically? How is your environment set up?

[rayway30419]

Hi @awood45

It happens randomly,

What kind of enviroment setting?

Ruby 2.3.0
Rails 5.0.0.1
aws-sdk 2.6.3
Ubuntu 14.04 Trusty (3.13.0-63-generic)

[awood45]

Is the Ubuntu version you listed your development environment, or an EC2 AMI? How are you configuring your clients in code?

[rayway30419]

It's on the EC2 AMI.

# region in configuration file
client = Aws::DynamoDB::Client.new(
  region: Aws.config[:region]
)
# do sth.
client.put_item params

[rayway30419]

BTW, I found a clue.....it's running on my sidekiq worker, it may happen to mult-ithreading?

[rayway30419]

Another error message when initialize DynamoDB client
NameError: uninitialized constant Aws::CredentialProviderChain::InstanceProfileCredentials

[awood45]

Okay, it looks like this may be related to how Sidekiq initializes - you might need to do eager_autoload within your Sidekiq workers, rather than only in the Rails initialization. Can you try that?

[awood45]

For what it's worth, we're working on an overhaul of the SDK which should reduce or eliminate these autoload issues - message definitely received that they cause problems when threading.

[rayway30419]

@awood45
With my understand, the sidekiq process will execute the procedures within rails initializer, too.

I am glad to hear this news.

[glennpratt]

https://bugs.ruby-lang.org/issues/12688

[trevorrowe]

Just chiming in here. The preview release of the modularized SDK is coming shortly. With modularization, we have eliminated the use of autoload.

[trevorrowe]

@rayway30419 Are you still experiencing the intermittent missing credential errors? Commonly this is caused by the aggressive defaults in attempting to load credentials from the instance metadata service. Once credentials expire, they must be refreshed, but if the metadata service fails to respond, a missing credentials error is raised. I've just push a small change that will go out with the next release where you can configure expanded retries and timeouts:

client = Aws::DynamoDB::Client.new(
  region: Aws.config[:region],
  instance_profile_credentials_timeout: 5, # defaults to 1 second
  instance_profile_credentials_retries: 5, # defaults to 0 retries
)

This change will be available in the next release.

[rayway30419]

hi @trevorrowe,
It still happens, I am looking forward to it!

[awood45]

It looks like this went out with a previous release. Let me know/reopen if you see this again after looking at configured retries/timeouts. Thanks!

[jammerful]

@trevorrowe I'm still seeing this issue on (actually coming from fluentd which uses the SDK):

Ruby 2.3.4
AWS SDK 2.7.0
Linux Kernel 4.9.41

Is this expected? Could this be eventual consistency, instead of code?