Add command to export credentials

[jamesls]

Describe the feature

This is attempting to drive closure to existing feature requests/PRs by proposing the exact feature set we should support for exporting credentials. This proposes an aws configure export-creds command that resolves the existing AWS CLI credential chain and formats the credentials in various formats, including as environment variables and in the credential process format, allowing the AWS CLI to be used as a credential process.

Use Case

For context, original issue:

• Provide way to get current credentials
• Current PR with partial implementation: Adding the option to export AWS credentials with a command #6808

To save you a few clicks, the rationale for this feature:

• While we strive for parity across our tools, there are times when the AWS CLI might support certain credential providers before other tools. Customers would like the ability to leverage the credential resolution process of the CLI in other SDKs and tools.
• Even with credential provider parity, there may be differences in implementations and configurations across tools. A common one that comes up, and is referenced in the linked issue, is that the CLI by default uses file based caching for credentials, where often times in SDKs you need to provide your own caching provider and explicitly provide the type of cache you want. Customers want a way want to reuse the existing caching mechanism/defaults in place in the CLI without having to configure each individual tool.
• Customers may sometimes need to interact with frameworks, tools, or services that only support a subset of credential providers, the most common one being environment variables. Customers would like the ability to export credentials in these formats, e.g. environment variables, to interop with other tools.
• There are several third party tools that provide this, or similar, functionality, but this is a crucial/core enough feature that we really should be providing a "built-in" mechanism that customers can utilize.
• This also gives you an easy way to consolidate all your credential related configuration in a single profile that you can use across multiple profiles. Admittedly, you can see this as a workaround for supporting "profile inheritance/mixins" in the config file (which has come up in various issues throughout the years), and I see that we've provided credential provider specific solutions to this (I see botocore has an sso_session section it supports), but it still provides a succinct way to reduce duplication in your AWS config file.

Proposed Solution

I propose we move forward with #6808 with a few additional changes.

• We will add an aws configure export-creds command that will use the AWS CLI's credential resolution process and output the results in various formats.
• The command will support a --format parameter that supports common formats the customers use for exporting credentials.
• The command will support outputting credentials in the output required by the credential process feature. This enables the AWS CLI to be used as a credential process.

Synopsis

aws configure export credentials [--format process|env|powershell|windows-cmd]

The default value will be process which outputs the credentials in the output required by a credential process.
We'll support the following output formats, which are the formats we see with SSO, or other console related tooling that allows you to export creds in various formats:

• process - The default output format, enables usage as a credential_process source.
• env - Exports env vars as in a posix compatible shell (export AWS_ACCESS_KEY_ID=...)
• powershell - Export env vars in powershell specific format ($Env:AWS_ACCESS_KEY_ID="...")
• windows-cmd - Export env vars in a windows cmd console format (set AWS_ACCESS_KEY_ID=...)

Examples

To use this command as a credential process in your config file:

[profile my-sso-config]
#  All the relevant sso config

[profile dev]
region = us-west-2
# Other non-cred related configuration
credential_process = aws configure export-creds --profile my-sso-config

[profile staging]
region = us-west-2
# Other non-cred related config for staging
credential_process = aws configure export-creds --profile my-sso-config

Then use just like any other profile:

$ aws s3 ls --profile dev
$ aws s3 ls --profile staging

To use this with env vars in a posix shell (bash/zsh etc):

$ eval $(aws configure export-creds --profile dev)
$ aws configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     ****************ABCD              env       # <----- Using the sourced env vars
secret_key     ****************EFGH              env       # <----- Using the sourced env vars
    region                us-west-2      config-file    ~/.aws/config

I noticed we have a few tools that also export fish shell creds, not sure if it's worth supporting at the moment so I've omitted it for now, but additional formatters would be straightforward to add. Also not supporting the "CLI config" format because if you're going to write out a new profile/section to the CLI config file, we should be recommending them to use a credential_process instead, which can properly handle (non-trivial) refreshes.

Other Information

One of the implementation details we'll need to figure out is being able to provide the expiry time of credentials, which is required by the credential process (if you want to refresh credentials), and arguably should be supported by other formats as well (e.g. env vars). For the AWS CLI, and specifically botocore, we do not provide the expiry time as a public attribute.

FWIW, I don't think I had any specific reason for not exposing the expiry time other than it was an implementation detail that wasn't relevant to consumers of credentials (I don't remember the exact timeline but I know we added assume role / refreshable credentials years before I introduced the credential process functionality). We really only cared about expiry time to handle the bookkeeping for advisory/mandatory refreshes.

I've seen the discussion on boto/botocore#2694, and while I think it is curious about the "not expired but maybe actually it is" expiry time specific to IMDS, I will follow up and see if there's any path forward there. Maybe we just add it to ReadOnlyCredentials with an extra attribute denoting expiry time fuzziness is sufficient.

Other implementation note we'll need to be careful of is detecting recursive invocations when running CLI commands against profiles that also have the AWS CLI configured as a credential process. We'll probably need to require you specify a different profile than the parent CLI process otherwise we'll error out, but we can explore what options we have available to us.

Last thing, I'm happy to drive this to completion and implement the remaining delta we need to get this feature complete.

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CLI version used

v2

Environment details (OS name and version, etc.)

All of them

[kyleknap]

This looks good. Just jotting down some thoughts:

1. Related to this comment:

   For the AWS CLI, and specifically botocore, we do not provide the expiry time as a public attribute.

   I'm fine with us making that public. For v2, we can always change it in our codebase and then backport it to botocore. It would be nice if this was a public attribute for this implementation.

2. One difference that I noticed between the process format and the rest of the formats is that the process format will include an expiration time while the environment variable related formats will not include an expiration time. Is there a particular reason the AWS_CREDENTIAL_EXPIRATION is not being set? I still need to think more on how that env var would potentially be used but was just something I noticed.

3. I'm wondering if it makes sense to have a simplified environment format with the export removed e.g.:

   aws configure export-creds --format env
   AWS_ACCESS_KEY_ID=<access>
   AWS_SECRET_ACCESS_KEY=<secret>
   AWS_SESSION_TOKEN=<token>
   

   Looking at aws-export-credentials by @benkehoe, it has that format: https://github.com/benkehoe/aws-export-credentials#env-vars. Also seems like the output could be a little easier to parse in the case you want to reformat the output to a format that we don't support. I'm mainly curious if you had considered it at all.

4. In doing recursive profile detection, the current approach seems fine in bailing out after X number of profiles (currently 4) have been traversed. Its difficult to imagine needing to recurse through more than that and we can always increase the threshold.

[benkehoe]

@kyleknap I think you might have intended this comment to be put on on the PR #7398, rather than the issue?

[kyleknap]

@benkehoe I meant this issue as originally we were trying to treat this issue as a medium for outlining the design/interface. I don't have a strong preference though as I will eventually be leaving comments in the PR.

[jamesls]

I'm fine with us making that public. For v2, we can always change it in our codebase and then backport it to botocore. It would be nice if this was a public attribute for this implementation.

If you want to go with that approach, it'll probably just be making the expiry time attribute public. I sketched out my ideal way to fix the issue in the linked botocore issue, tl;dr move the fake expiry logic up a layer into the IMDS credential provider, but it's a more invasive change that will need a good amount of testing (assuming it's even possible) so I'd suggest holding off on that larger change for now.

Is there a particular reason the AWS_CREDENTIAL_EXPIRATION is not being set?

Nice! I completely forgot that we supported that (I even mentioned in this issue we should support refreshing from env vars). I'll update.

Looking at aws-export-credentials by @benkehoe, it has that format: https://github.com/benkehoe/aws-export-credentials#env-vars. Also seems like the output could be a little easier to parse in the case you want to reformat the output to a format that we don't support. I'm mainly curious if you had considered it at all.

Hadn't considered it, it's never come up in my usage. I wouldn't mind adding it, assuming we had a use case for it, but I like keeping the env format for the exported version. It's what you get from the sso console and all the other tools/consoles we use to get creds you can use in your shell so I think there's an expectation that this is what an environment variable format should be. Suggestions on the name?

[benkehoe]

Suggestions on the name?

aws-export-credentials has --env and --env-export

a more invasive change

Over on the issue I suggest making expiry time public but not setting it for IMDS credentials for now, allowing the change to be made quicker, not waiting for the larger movement of the logic to happen.

[jaklan]

Hi guys,

I see #7398 is merged and released, so the issue probably should be closed, but I noticed one strange behaviour:

❯ AWS_PROFILE=some-profile aws configure export-credentials --format env
export AWS_ACCESS_KEY_ID=[redacted]
export AWS_SECRET_ACCESS_KEY=[redacted]
export AWS_SESSION_TOKEN=[redacted]
export AWS_CREDENTIAL_EXPIRATION=2023-01-01T18:18:38+00:00

that works as expected, but - when I use granted to assume role:

❯ aws-assume some-profile
[✔] [some-profile](eu-central-1) session credentials will expire in 8 hours

❯ aws configure export-credentials --format env
export AWS_ACCESS_KEY_ID=[redacted]
export AWS_SECRET_ACCESS_KEY=[redacted]
export AWS_SESSION_TOKEN=[redacted]

the AWS_CREDENTIAL_EXPIRATION variable is missing.

What's interesting - it works correctly with @benkehoe's aws-export-credentials in both scenarios:

❯ aws-assume some-profile
[✔] [some-profile](eu-central-1) session credentials will expire in 8 hours

❯ aws-export-credentials --env-export
export AWS_ACCESS_KEY_ID=[redacted]
export AWS_SECRET_ACCESS_KEY=[redacted]
export AWS_SESSION_TOKEN=[redacted]
export AWS_CREDENTIALS_EXPIRATION=2023-01-01T18:28:15Z

[jaklan]

Ah okay, I guess I got the point - when envars are set, aws configure export-credentials just utilise them. So AWS_CREDENTIAL_EXPIRATION variable has to be set by granted (or any other tool) and then aws configure export-credentials would work correctly:

❯ export AWS_CREDENTIAL_EXPIRATION=2023-01-01T19:54:27+01:00

❯ aws configure export-credentials --format env
export AWS_ACCESS_KEY_ID=[redacted]
export AWS_SECRET_ACCESS_KEY=[redacted]
export AWS_SESSION_TOKEN=[redacted]
export AWS_CREDENTIAL_EXPIRATION=2023-01-01T19:54:27+01:00

That was fine with aws-export-credentials, because it works a bit differently:

There are a number of other projects that extract AWS credentials and/or inject them into programs, but all the ones I've seen use the CLI's cache files directly, rather than leveraging botocore's ability to retrieve and refresh credentials. So I wrote this to do that.

[tim-finnigan]

Since #7398 was merged as mentioned above, I also think this issue can be closed. A new issue can be created to track any issues related to this one going forward.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.