credentials mocking

[sean-abbott]

This is either a question or a request.

I've been unable to find any references to credentials profiles in the code, so I'm guessing these aren't implemented yet.

I'm using different profiles for different groups, and I need to make my s3 connection using profile (i.e., aws_session = boto3.Session(profile_name=profile_name)

moto does not appear to mock the profile name; it must exist in the developers ~/.aws/credentials (or config).

Any chance this can be implemented and that other folks need it?

[spulec]

When you say moto does not appear to mock the profile name, what are you seeing to indicate that? Are you making some IAM calls? For the most part, Moto ignores all of the credentials as of now.

If you could add some code, that might help me understand the use case.

[sean-abbott]

I've since forgotten the specifics, but I think it comes down to calling boto3.Session with a profile or attempting to use it caused it to look for (and not find) the ~/.aws/credentials|config file.

I fixed by manually mocking the call expanduser call.

If I am doing this poorly, please do let me know (especially if you can suggest a fix!) I'm not great at testing.

https://github.com/sean-abbott/terraform.py/blob/master/tests/remote/test_s3.py

[spulec]

Hmmmm. What happens if you don't mock the credentials?

[sean-abbott]

botocore.exceptions.ProfileNotFound: The config profile (testawsprofile) could not be found

call is in test_get_remote_state, calling ati.remote.s3_remote_state.get_remote_state (line 69)

s3_remote_state.py:36: aws_session = boto3.Session(profile_name=profile_name

This happens if I comment line 60 in tests/remote/test_s3.py (which is where I patch expanduser)

Here's a branch which demos the issue. You can run py.test tests/remote/test_s3.py after doing a pip install .[test]. Uncommenting the monkeypatch makes it work again.
https://github.com/sean-abbott/terraform.py/tree/test_no_credentials

[JackDanger]

@sean-abbott Apologies, I tried to find where you were mocking the credentials but I'm not sure I found the right place. Is it this?: sean-abbott/terraform.py@d2ed646

[JackDanger]

@sean-abbott I'm going to close this for now but please reopen when you get the chance.

[dan-coates]

I think I've come across this same issue and have a relatively easy way to reproduce it. Essentially, if the AWS_PROFILE environment variable is set but the profile isn't found in ~/.aws/config, it raises an exception on import:

________________________________________ ERROR collecting tests/test_create_cluster.py _________________________________________
tests/test_create_cluster.py:2: in <module>
    from moto import mock_s3
venv/lib/python3.6/site-packages/moto/__init__.py:9: in <module>
    from .apigateway import mock_apigateway, mock_apigateway_deprecated  # flake8: noqa
venv/lib/python3.6/site-packages/moto/apigateway/__init__.py:2: in <module>
    from .models import apigateway_backends
venv/lib/python3.6/site-packages/moto/apigateway/models.py:648: in <module>
    for region_name in Session().get_available_regions('apigateway'):
venv/lib/python3.6/site-packages/boto3/session.py:80: in __init__
    self._setup_loader()
venv/lib/python3.6/site-packages/boto3/session.py:120: in _setup_loader
    self._loader = self._session.get_component('data_loader')
venv/lib/python3.6/site-packages/botocore/session.py:729: in get_component
    return self._components.get_component(name)
venv/lib/python3.6/site-packages/botocore/session.py:946: in get_component
    self._components[name] = factory()
venv/lib/python3.6/site-packages/botocore/session.py:186: in <lambda>
    lambda:  create_loader(self.get_config_variable('data_path')))
venv/lib/python3.6/site-packages/botocore/session.py:281: in get_config_variable
    elif self._found_in_config_file(methods, var_config):
venv/lib/python3.6/site-packages/botocore/session.py:308: in _found_in_config_file
    return var_config[0] in self.get_scoped_config()
venv/lib/python3.6/site-packages/botocore/session.py:385: in get_scoped_config
    raise ProfileNotFound(profile=profile_name)
E   botocore.exceptions.ProfileNotFound: The config profile (~~~my-aws-profile-name~~~) could not be found

If I unset AWS_PROFILE, it's totally fine with there being no ~/.aws/config at all. Since it's fine if config doesn't exist, it makes me think that it should tolerate a specific profile not being there.

[drorata]

I suspect I'm witnessing a related, but different, issue. When I run the tests locally everything is fine (and I have the AWS credentials in ~/.aws). Once I try to run the tests within a docker container (which doesn't contain the credentials) I get the following error:

botocore.exceptions.NoCredentialsError: Unable to locate credentials

This is thrown right after appearances of:

s3_conn = boto3.client('s3', 'us-west-2')
s3_conn.create_bucket(Bucket='test-bucket')

If, when I start the container, I mount my credentials the tests all pass. Furthermore, creating a dummy ~/.aws/credentials with the values:

[default]
aws_access_key_id = fOo
aws_secret_access_key = bAr

is enough and the tests passes.

NB: I'm not sure if it is related, but I have update some dependencies:

-moto==1.3.6
-boto3==1.7.84
-botocore==1.10.84
+moto==1.3.7
+boto3==1.9.38
+botocore>=1.12.13

This might be related.