Credentials issue introduced in 0.27.0 when running on EC2 instance. #163

mjharvey · 2023-05-02T20:35:42Z

Hi. I have been using awscurl on an EC2 instance to call an API endpoint that is secured via IAM permissions.
This worked fine under 0.26.0, but with the release of 0.27.0 I am now encountering this error

  File "/usr/local/bin/awscurl", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/awscurl/awscurl.py", line 543, in main
    inner_main(sys.argv[1:])
  File "/usr/local/lib/python3.6/site-packages/awscurl/awscurl.py", line 508, in inner_main
    args.profile)
  File "/usr/local/lib/python3.6/site-packages/awscurl/awscurl.py", line 427, in load_aws_config
    cred = session.get_credentials()
  File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 449, in get_credentials
    'credential_provider').load_credentials()
  File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 897, in get_component
    self._components[name] = factory()
  File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 176, in <lambda>
    lambda:  botocore.credentials.create_credential_resolver(self))
  File "/usr/local/lib/python3.6/site-packages/botocore/credentials.py", line 55, in create_credential_resolver
    metadata_timeout = session.get_config_variable('metadata_service_timeout')
  File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 265, in get_config_variable
    elif self._found_in_config_file(methods, var_config):
  File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 286, in _found_in_config_file
    return var_config[0] in self.get_scoped_config()
  File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 358, in get_scoped_config
    raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile (default) could not be found

The problem looks like it might have been caused by this change: 7b38c7f?diff=split?diff=split?diff=split%3Fdiff%3Dsplit

Is this a known issue, or this there a workaround for the problem I could apply?

Thanks.

The text was updated successfully, but these errors were encountered:

okigan · 2023-05-02T21:01:29Z

Let me know how you invoke awscurl, and also how do you specify the credentials (environment variable, command line, etc). And thanks for reporting the issue.

…

On Tue, May 2, 2023 at 3:35 PM Matt Harvey ***@***.***> wrote: Hi. I have been using awscurl on an EC2 instance to call an API endpoint that is secured via IAM permissions. This worked fine under 0.26.0, but with the release of 0.27.0 I am now encountering this error File "/usr/local/bin/awscurl", line 11, in <module> sys.exit(main()) File "/usr/local/lib/python3.6/site-packages/awscurl/awscurl.py", line 543, in main inner_main(sys.argv[1:]) File "/usr/local/lib/python3.6/site-packages/awscurl/awscurl.py", line 508, in inner_main args.profile) File "/usr/local/lib/python3.6/site-packages/awscurl/awscurl.py", line 427, in load_aws_config cred = session.get_credentials() File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 449, in get_credentials 'credential_provider').load_credentials() File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 897, in get_component self._components[name] = factory() File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 176, in <lambda> lambda: botocore.credentials.create_credential_resolver(self)) File "/usr/local/lib/python3.6/site-packages/botocore/credentials.py", line 55, in create_credential_resolver metadata_timeout = session.get_config_variable('metadata_service_timeout') File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 265, in get_config_variable elif self._found_in_config_file(methods, var_config): File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 286, in _found_in_config_file return var_config[0] in self.get_scoped_config() File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 358, in get_scoped_config raise ProfileNotFound(profile=profile_name) botocore.exceptions.ProfileNotFound: The config profile (default) could not be found The problem looks like it might have been caused by this change: 7b38c7f ?diff=split?diff=split?diff=split%3Fdiff%3Dsplit <7b38c7f?diff=split?diff=split?diff=split%3Fdiff%3Dsplit> Is this a known issue, or this there a workaround for the problem I could apply? Thanks. — Reply to this email directly, view it on GitHub <#163>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADUYXVYESPMQBZQX2BWT3TXEFV2RANCNFSM6AAAAAAXTRIPQM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

mjharvey · 2023-05-02T23:22:21Z

There are no environment variables or profile setup on the EC2 instance for authentication. What I am guessing was happening previously was the session = botocore.session.get_session() gets credentials for the execution role associated with the EC2 instance (I may be wrong though).

okigan · 2023-05-02T23:41:50Z

Alright I’ll recheck on ec2 instance. Which service are you accessing (ec2, lambda..)?On May 2, 2023, at 6:22 PM, Matt Harvey ***@***.***> wrote: There are no environment variables or profile setup on the EC2 instance for authentication. What I am guessing was happening previously was the session = botocore.session.get_session() gets credentials for the execution role associated with the EC2 instance (I may be wrong though). —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

mjharvey · 2023-05-03T00:11:03Z

Lambda via API Gateway endpoint. The API gateway endpoint is secured via IAM policy that restricts access based on AWS Organisation Units.

…

________________________________ From: Igor Okulist ***@***.***> Sent: Wednesday, 3 May 2023 11:42 am To: okigan/awscurl ***@***.***> Cc: Matt Harvey ***@***.***>; Author ***@***.***> Subject: Re: [okigan/awscurl] Credentials issue introduced in 0.27.0 when running on EC2 instance. (Issue #163) Alright I’ll recheck on ec2 instance. Which service are you accessing (ec2, lambda..)?On May 2, 2023, at 6:22 PM, Matt Harvey ***@***.***> wrote: There are no environment variables or profile setup on the EC2 instance for authentication. What I am guessing was happening previously was the session = botocore.session.get_session() gets credentials for the execution role associated with the EC2 instance (I may be wrong though). —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***> — Reply to this email directly, view it on GitHub<#163 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABHSEFSDHKDNVKBKOIMUKVDXEGLUTANCNFSM6AAAAAAXTRIPQM>. You are receiving this because you authored the thread.Message ID: ***@***.***>

okigan · 2023-05-03T21:25:08Z

got pulled in with other items -- will review asap.

okigan · 2023-05-05T15:26:39Z

Capturing nice way to repro in aws cloud shell:

[cloudshell-user@ip-10-4-127-13 ~]$ pip3 install -q awscurl==0.26
[cloudshell-user@ip-10-4-127-13 ~]$ awscurl --service s3 https://awscurl-sample-bucket.s3.amazonaws.com 
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>awscurl-sample-bucket</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>awscurl-sample-file.txt</Key><LastModified>2017-07-25T21:27:38.000Z</LastModified><ETag>&quot;d41d8cd98f00b204e9800998ecf8427e&quot;</ETag><Size>0</Size><Owner><ID>978427f78b00827efacb8fe2bd55ea30cbcb1d228cd3758972314cb67e763402</ID><DisplayName>okigan</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>



[cloudshell-user@ip-10-4-127-13 ~]$ pip3 install -q awscurl==0.27
[cloudshell-user@ip-10-4-127-13 ~]$ awscurl --service s3 https://awscurl-sample-bucket.s3.amazonaws.com 
Traceback (most recent call last):
  File "/home/cloudshell-user/.local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/awscurl/awscurl.py", line 543, in main
    inner_main(sys.argv[1:])
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/awscurl/awscurl.py", line 508, in inner_main
    args.profile)
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/awscurl/awscurl.py", line 427, in load_aws_config
    cred = session.get_credentials()
  File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 508, in get_credentials
    'credential_provider'
  File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 1108, in get_component
    self._components[name] = factory()
  File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 187, in _create_credential_resolver
    self, region_name=self._last_client_region_used
  File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 74, in create_credential_resolver
    metadata_timeout = session.get_config_variable('metadata_service_timeout')
  File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 318, in get_config_variable
    logical_name
  File "/usr/local/lib/python3.7/site-packages/botocore/configprovider.py", line 426, in get_config_variable
    return provider.provide()
  File "/usr/local/lib/python3.7/site-packages/botocore/configprovider.py", line 628, in provide
    value = provider.provide()
  File "/usr/local/lib/python3.7/site-packages/botocore/configprovider.py", line 718, in provide
    scoped_config = self._session.get_scoped_config()
  File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 416, in get_scoped_config
    raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile (default) could not be found
[cloudshell-user@ip-10-4-127-13 ~]$

okigan · 2023-05-05T17:01:16Z

Output from the latest release:

[cloudshell-user@ip-10-2-12-122 ~]$ pip3 install -q awscurl==0.28
[cloudshell-user@ip-10-2-12-122 ~]$ awscurl --service s3 https://awscurl-sample-bucket.s3.amazonaws.com 
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>awscurl-sample-bucket</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>awscurl-sample-file.txt</Key><LastModified>2017-07-25T21:27:38.000Z</LastModified><ETag>&quot;d41d8cd98f00b204e9800998ecf8427e&quot;</ETag><Size>0</Size><Owner><ID>978427f78b00827efacb8fe2bd55ea30cbcb1d228cd3758972314cb67e763402</ID><DisplayName>okigan</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>awscurl-sample-file:.txt</Key><LastModified>2023-05-05T16:03:34.000Z</LastModified><ETag>&quot;d41d8cd98f00b204e9800998ecf8427e&quot;</ETag><Size>0</Size><Owner><ID>978427f78b00827efacb8fe2bd55ea30cbcb1d228cd3758972314cb67e763402</ID><DisplayName>okigan</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>

okigan added a commit that referenced this issue May 5, 2023

Resolve #163

8493c61

okigan mentioned this issue May 5, 2023

Resolve https://github.com/okigan/awscurl/issues/163 #166

Merged

okigan closed this as completed in #166 May 5, 2023

okigan added a commit that referenced this issue May 5, 2023

Resolve #163 (#166)

f641e98

okigan mentioned this issue May 5, 2023

Request: Support for Container credentails #164

Closed

erpel mentioned this issue May 11, 2023

Does not support AssumeRole or MFA profiles #44

Open

erpel mentioned this issue May 25, 2023

Use profile if set, remove default #172

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Credentials issue introduced in 0.27.0 when running on EC2 instance. #163

Credentials issue introduced in 0.27.0 when running on EC2 instance. #163

mjharvey commented May 2, 2023

okigan commented May 2, 2023 via email

mjharvey commented May 2, 2023

okigan commented May 2, 2023 via email

mjharvey commented May 3, 2023 via email

okigan commented May 3, 2023

okigan commented May 5, 2023

okigan commented May 5, 2023 •

edited

Loading

Credentials issue introduced in 0.27.0 when running on EC2 instance. #163

Credentials issue introduced in 0.27.0 when running on EC2 instance. #163

Comments

mjharvey commented May 2, 2023

okigan commented May 2, 2023 via email

mjharvey commented May 2, 2023

okigan commented May 2, 2023 via email

mjharvey commented May 3, 2023 via email

okigan commented May 3, 2023

okigan commented May 5, 2023

okigan commented May 5, 2023 • edited Loading

okigan commented May 5, 2023 •

edited

Loading