Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with credential finding algorithm #2387

Closed
ryanisnan opened this issue May 13, 2021 · 10 comments
Closed

Issue with credential finding algorithm #2387

ryanisnan opened this issue May 13, 2021 · 10 comments
Assignees
Labels
closed-for-staleness guidance Question that needs advice or information. needs-reproduction response-requested Waiting on additional info and feedback.

Comments

@ryanisnan
Copy link

ryanisnan commented May 13, 2021

Describe the bug
botocore is unable to locate credentials from either the SSO flow or the shared credentials file methods.

Additional information
The steps below are only for the SSO credential flow, but I also was able to reproduce this bug using the shared credentials file flow. The environment variable method of credential access did work however.

Steps to reproduce

  1. Upgrade boto3 to 1.17.72
  2. Upgrade botocore to 1.20.72
  3. Ensure you have an SSO profile configured, e.g.:
[profile deploy]
sso_start_url = https://acme.awsapps.com/start
sso_region = us-east-1
sso_account_id = 8675309
sso_role_name = DeploymentAccess
region = us-west-1
output = json
  1. Log in to the CLI using the SSO flow, e.g. aws sso login --profile deploy
  2. Attempt to use the library, e.g.:
client = boto3.client("opsworks", region_name=region_name)
stacks = client.describe_stacks().get("Stacks", [])
  1. An exception is raised stating: botocore.exceptions.NoCredentialsError: Unable to locate credentials (see full stack trace below)

Expected behavior
Results should be returned in the event the found credentials have the required permissions, or we get rejected on the basis of insufficient permissions.

Debug logs

  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/./app.py", line 660, in <module>
    main()
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/./app.py", line 638, in main
    lambda_handler(
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/./app.py", line 556, in lambda_handler
    stack = get_stack(stack_name, region)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/./app.py", line 166, in get_stack
    stacks = client.describe_stacks().get("Stacks", [])
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/client.py", line 662, in _make_api_call
    http, parsed_response = self._make_request(
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/client.py", line 682, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/endpoint.py", line 115, in create_request
    self._event_emitter.emit(event_name, request=request,
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/signers.py", line 162, in sign
    auth.add_auth(request)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/auth.py", line 373, in add_auth
    raise NoCredentialsError()
botocore.exceptions.NoCredentialsError: Unable to locate credentials
@ryanisnan ryanisnan added the needs-triage This issue or PR still needs to be triaged. label May 13, 2021
@kdaily kdaily self-assigned this May 14, 2021
@kdaily kdaily added investigating This issue is being investigated and/or work is in progress to resolve the issue. and removed needs-triage This issue or PR still needs to be triaged. labels May 14, 2021
@kdaily
Copy link
Member

kdaily commented May 14, 2021

Hi @ryanisnan,

With regards to the issues with SSO, I think you're running into this issue:

#1923

There is a detailed response on the design choices for this already, so I won't duplicate it here:

#1923 (comment)

With regard to not getting credentials from the shared credential file, can you provide some debug logs and your example, please? Thanks!

@kdaily kdaily added guidance Question that needs advice or information. response-requested Waiting on additional info and feedback. investigating This issue is being investigated and/or work is in progress to resolve the issue. and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels May 14, 2021
@kdaily
Copy link
Member

kdaily commented May 14, 2021

@ryanisnan,

I tried to reproduce as well and I was successful with logging in to SSO via the AWS CLI V2 and then using the same profile to access AWS services. Can you provide any further debug logs using boto3.set_stream_logger('')?

@kdaily kdaily added response-requested Waiting on additional info and feedback. and removed response-requested Waiting on additional info and feedback. investigating This issue is being investigated and/or work is in progress to resolve the issue. labels May 14, 2021
@ryanisnan
Copy link
Author

Hey @kdaily thanks for looking into this!

Here's what added that logging spits out:

❯ ./app.py --branch 20210518 --environment stage --dry-run
2021-05-14 17:09:33,481 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2021-05-14 17:09:33,483 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2021-05-14 17:09:33,484 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2021-05-14 17:09:33,485 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2021-05-14 17:09:33,486 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2021-05-14 17:09:33,486 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2021-05-14 17:09:33,487 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2021-05-14 17:09:33,490 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2021-05-14 17:09:33,490 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2021-05-14 17:09:33,490 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2021-05-14 17:09:33,490 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2021-05-14 17:09:33,497 botocore.utils [DEBUG] IMDS ENDPOINT: http://169.254.169.254/
2021-05-14 17:09:33,508 botocore.credentials [DEBUG] Looking for credentials via: env
2021-05-14 17:09:33,508 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2021-05-14 17:09:33,508 botocore.credentials [DEBUG] Looking for credentials via: assume-role-with-web-identity
2021-05-14 17:09:33,508 botocore.credentials [DEBUG] Looking for credentials via: sso
2021-05-14 17:09:33,508 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2021-05-14 17:09:33,509 botocore.credentials [DEBUG] Looking for credentials via: custom-process
2021-05-14 17:09:33,509 botocore.credentials [DEBUG] Looking for credentials via: config-file
2021-05-14 17:09:33,510 botocore.credentials [DEBUG] Looking for credentials via: ec2-credentials-file
2021-05-14 17:09:33,510 botocore.credentials [DEBUG] Looking for credentials via: boto-config
2021-05-14 17:09:33,510 botocore.credentials [DEBUG] Looking for credentials via: container-role
2021-05-14 17:09:33,510 botocore.credentials [DEBUG] Looking for credentials via: iam-role
2021-05-14 17:09:33,511 urllib3.connectionpool [DEBUG] Starting new HTTP connection (1): 169.254.169.254:80
2021-05-14 17:09:34,515 botocore.utils [DEBUG] Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/api/token: Connect timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
Traceback (most recent call last):
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/httpsession.py", line 314, in send
    urllib_response = conn.urlopen(
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1253, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/awsrequest.py", line 92, in _send_request
    rval = super(AWSConnection, self)._send_request(
  File "/usr/local/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1299, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1248, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/awsrequest.py", line 120, in _send_output
    self.send(msg)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/awsrequest.py", line 204, in send
    return super(AWSConnection, self).send(str)
  File "/usr/local/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 948, in send
    self.connect()
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/connection.py", line 200, in connect
    conn = self._new_conn()
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPConnection object at 0x102d2b430>, 'Connection to 169.254.169.254 timed out. (connect timeout=1)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/utils.py", line 381, in _fetch_metadata_token
    response = self._session.send(request.prepare())
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/httpsession.py", line 347, in send
    raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
2021-05-14 17:09:34,520 urllib3.connectionpool [DEBUG] Starting new HTTP connection (2): 169.254.169.254:80
2021-05-14 17:09:35,522 botocore.utils [DEBUG] Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/iam/security-credentials/: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
Traceback (most recent call last):
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/httpsession.py", line 314, in send
    urllib_response = conn.urlopen(
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1253, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/awsrequest.py", line 92, in _send_request
    rval = super(AWSConnection, self)._send_request(
  File "/usr/local/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1299, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1248, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/awsrequest.py", line 120, in _send_output
    self.send(msg)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/awsrequest.py", line 204, in send
    return super(AWSConnection, self).send(str)
  File "/usr/local/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 948, in send
    self.connect()
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/connection.py", line 200, in connect
    conn = self._new_conn()
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPConnection object at 0x102d2ba60>, 'Connection to 169.254.169.254 timed out. (connect timeout=1)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/utils.py", line 429, in _get_request
    response = self._session.send(request.prepare())
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/httpsession.py", line 347, in send
    raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
2021-05-14 17:09:35,523 botocore.utils [DEBUG] Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.
2021-05-14 17:09:35,524 botocore.loaders [DEBUG] Loading JSON file: /Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/data/endpoints.json
2021-05-14 17:09:35,531 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x102be49d0>
2021-05-14 17:09:35,584 botocore.loaders [DEBUG] Loading JSON file: /Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/data/opsworks/2013-02-18/service-2.json
2021-05-14 17:09:35,590 botocore.hooks [DEBUG] Event creating-client-class.opsworks: calling handler <function add_generate_presigned_url at 0x102b8c700>
2021-05-14 17:09:35,592 botocore.endpoint [DEBUG] Setting opsworks timeout as (60, 60)
2021-05-14 17:09:35,593 botocore.loaders [DEBUG] Loading JSON file: /Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/data/_retry.json
2021-05-14 17:09:35,594 botocore.client [DEBUG] Registering retry handlers for service: opsworks
2021-05-14 17:09:35,595 botocore.hooks [DEBUG] Event before-parameter-build.opsworks.DescribeStacks: calling handler <function generate_idempotent_uuid at 0x102c0bc10>
2021-05-14 17:09:35,595 botocore.hooks [DEBUG] Event before-call.opsworks.DescribeStacks: calling handler <function inject_api_version_header_if_needed at 0x102c134c0>
2021-05-14 17:09:35,595 botocore.endpoint [DEBUG] Making request for OperationModel(name=DescribeStacks) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'OpsWorks_20130218.DescribeStacks', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'Boto3/1.17.73 Python/3.9.5 Darwin/20.1.0 Botocore/1.20.73'}, 'body': b'{}', 'url': 'https://opsworks.us-west-1.amazonaws.com/', 'context': {'client_region': 'us-west-1', 'client_config': <botocore.config.Config object at 0x1092c5550>, 'has_streaming_input': False, 'auth_type': None}}
2021-05-14 17:09:35,595 botocore.hooks [DEBUG] Event request-created.opsworks.DescribeStacks: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x1092c5370>>
2021-05-14 17:09:35,596 botocore.hooks [DEBUG] Event choose-signer.opsworks.DescribeStacks: calling handler <function set_operation_specific_signer at 0x102c0baf0>
Traceback (most recent call last):
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/./app.py", line 662, in <module>
    main()
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/./app.py", line 632, in main
    validate_arguments(arguments, stacks_to_deploy)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/./app.py", line 530, in validate_arguments
    current_app_date = get_app_url_date(stacks[0])
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/./app.py", line 503, in get_app_url_date
    app = get_app(get_stack(stack["name"], stack["region"]), stack["app"], stack["region"])
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/./app.py", line 168, in get_stack
    stacks = client.describe_stacks().get("Stacks", [])
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/client.py", line 691, in _make_api_call
    http, parsed_response = self._make_request(
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/client.py", line 711, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/endpoint.py", line 115, in create_request
    self._event_emitter.emit(event_name, request=request,
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/signers.py", line 162, in sign
    auth.add_auth(request)
  File "/Users/ryanwest/Code/el8/infrastructure/legacy/src/venv/lib/python3.9/site-packages/botocore/auth.py", line 373, in add_auth
    raise NoCredentialsError()
botocore.exceptions.NoCredentialsError: Unable to locate credentials

I'm not sure why it's trying to make any calls to the metadata API, I'm not on ec2...

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. label May 15, 2021
@kdaily
Copy link
Member

kdaily commented May 17, 2021

Thanks for the debug, I can see that it is skipping over all of the credential methods in the chain, and falling back to IMDS:

2021-05-14 17:09:33,508 botocore.credentials [DEBUG] Looking for credentials via: env
2021-05-14 17:09:33,508 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2021-05-14 17:09:33,508 botocore.credentials [DEBUG] Looking for credentials via: assume-role-with-web-identity
2021-05-14 17:09:33,508 botocore.credentials [DEBUG] Looking for credentials via: sso
2021-05-14 17:09:33,508 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2021-05-14 17:09:33,509 botocore.credentials [DEBUG] Looking for credentials via: custom-process
2021-05-14 17:09:33,509 botocore.credentials [DEBUG] Looking for credentials via: config-file
2021-05-14 17:09:33,510 botocore.credentials [DEBUG] Looking for credentials via: ec2-credentials-file
2021-05-14 17:09:33,510 botocore.credentials [DEBUG] Looking for credentials via: boto-config
2021-05-14 17:09:33,510 botocore.credentials [DEBUG] Looking for credentials via: container-role
2021-05-14 17:09:33,510 botocore.credentials [DEBUG] Looking for credentials via: iam-role

Have you moved the location of the shared credentials file, or is it in the default location (looks like it should be /Users/ryanwest/.aws from your debug logs)?

Can you successfully use any other method, like environment variables, successfully?

@kdaily kdaily added the response-requested Waiting on additional info and feedback. label May 17, 2021
@ryanisnan
Copy link
Author

ryanisnan commented May 18, 2021 via email

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. label May 18, 2021
@kdaily
Copy link
Member

kdaily commented Jun 7, 2021

Thanks @ryanisnan. I'm not seeing anything obvious.

  1. Can you successfully run another AWS CLI command after logging in to SSO?
  2. Can you confirm your installation method? I see some references to Homebrew-installed infrastructure, and you're using Python 3.9 (not technically supported, but is not usually an issue).

@kdaily kdaily added the response-requested Waiting on additional info and feedback. label Jun 7, 2021
@kdaily
Copy link
Member

kdaily commented Jun 8, 2021

@ryanisnan,

Maybe I overlooked something simple. You would need to specify the same profile with a boto3.Session in app.py. Based on your example you've used the default boto3 session. It would look something like this:

import boto3
session = boto3.Session(profile_name='deploy')
client = session.client("opsworks")
stacks = client.describe_stacks().get("Stacks", [])

@kdaily kdaily added response-requested Waiting on additional info and feedback. and removed response-requested Waiting on additional info and feedback. labels Jun 8, 2021
@github-actions
Copy link

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

@ConnorJBeck
Copy link

If anyone finds this in the future, my problem was basically what @kdaily last posted - boto3 wasn't using the profile/credentials I was expecting because I hadn't specified any particular profile/credentials. It seems like there's a few simple ways to handle that (I'm sure there's more though):

  1. As he said, specify the profile when creating the session in the script.
  2. Set a default AWS Profile via the CLI as an environment variable, see the AWS Docs on using profiles with CLI.
  3. Configure an overall default profile in the config/credentials file that will be a bit more permanent, see the section on modifying your config/creds file for how to specify a default profile.

You could also probably set up your script to accept a profile and use the specified profile when starting the session, but that's a bit out of scope for this comment.

The tipoff for me was realizing I also couldn't execute aws cli commands that required authentication (e.g. aws s3 ls) without also specifying a profile (e.g. aws s3 ls --profile deploy). After setting a default profile, I no longer needed to specify a profile to get the command working, and my boto3 script started working too.

@yaoshiang
Copy link

I'm ran into this issue as well, with boto3 v 1.26.162 and botocore 1.29.162 and python 3.10.

I was using IAM identity center, which stores its config info in ~/.aws/config. According to boto3 docs, that file (and hence sso / IAM identity center) are searched on after ~/.aws/credentials. Hence, if you are using both IAM and IAM identity center (aka sso), aws will always search on the IAM profiles first. if you have a default specified, that's what gets used, not the IAM identity center (aka sso) configuration in ~/.aws/cconfig.

I deleted ~/.aws/credentials and still no luck.

I got it working by adding the --profile xyz flag after the aws command.

To set a default profile so I don't have to use the --profile xyz flag, or specify a profile in python using boto, I copied the sso_session, sso_account_id, sso_role_name, and region rows from the [profile xyz] section and pasted them under the [default] section.

Problem solved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
closed-for-staleness guidance Question that needs advice or information. needs-reproduction response-requested Waiting on additional info and feedback.
Projects
None yet
Development

No branches or pull requests

4 participants