Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Making the ConnectionManager a more "friendly" interface for hooks, resolvers, and template handlers #1287

Merged

Conversation

jfalkenstein
Copy link
Contributor

@jfalkenstein jfalkenstein commented Jan 8, 2023

The problem

When developing custom hooks, resolvers, and template_handlers, it is often necessary to invoke AWS services. The correct way to do this is via the stack's ConnectionManager so that they can work according to the project and stack configurations for AWS connections (particularly with regard to the profile, region, and iam_role).

I consider the ease of developing custom hooks, resolvers, and template_handlers one of Sceptre's great strengths that distinguishes it from other competing utilities. However, over my time developing these custom extensions, it has not often been straight-forward to invoke AWS services. I've noticed a few ugly bits that I have only been able to make work because of my fairly deep knowledge of the inner workings of Sceptre:

  • There isn't really any documentation on the correct way to do this stuff. It's likely that a lot of the custom extensions to Sceptre out there do these things the wrong way. The correct way to use boto3 (with regard to the Stack configs), for example, isn't really clear. We need better docs.
  • If you only need to make a simple call to a boto3 service method, you can use self.stack.connection_manager.call() and that works pretty well. It's still not documented, though.
  • HOWEVER, if you need to use a boto3 resource object, like an S3 Bucket resource, the only way to do this is via a private, undocumented method _get_session on the ConnectionManager. And to use that private method, you have to know when and how to use, override, or ignore the default ConnectionManager configurations. We need a public interface for getting a session.
  • If your hook/resolver/template handler needs to invoke a third-party tool via subprocess, the only way to get authentication to work correctly is to (1) get a session (see the second point above), then (2) use that to get credentials on that session, then (3) create environment variables that can be passed to Subprocess. The means of doing this correctly has taken me a lot of trial and error, isn't documented, and is rather gross. We need an easy way to get authenticated credential environment variables for our Stack Configs to integrate with third party tools.

A few examples of places I've had to do this recently, to demonstrate how "ugly" this process is can be found at:

How this PR fixes this:

  1. I include real docs on how to call AWS services and integrate with third party tools.
  2. I've created a public get_session() method that has the "smarts" to normally be used for default purposes with customizability.
  3. I've created a new method for creating the environment variables for the session for easy third party integration.

PR Checklist

  • Wrote a good commit message & description [see guide below].
  • Commit message starts with [Resolve #issue-number].
  • Added/Updated unit tests.
  • Added/Updated integration tests (if applicable).
  • All unit tests (make test) are passing.
  • Used the same coding conventions as the rest of the project.
  • The new code passes pre-commit validations (pre-commit run --all-files).
  • The PR relates to only one subject with a clear title.
    and description in grammatically correct, complete sentences.

Approver/Reviewer Checklist

  • Before merge squash related commits.

Other Information

Guide to writing a good commit

@jfalkenstein jfalkenstein marked this pull request as ready for review January 8, 2023 18:36
@jfalkenstein jfalkenstein self-assigned this Jan 8, 2023
@jfalkenstein jfalkenstein changed the title Making the ConnectionManager a more "friendly" interface for hooks, resolvers, and template_handlers Making the ConnectionManager a more "friendly" interface for hooks, resolvers, and template handlers Jan 8, 2023
Copy link
Contributor

@ericabrauer ericabrauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link
Contributor

@dboitnot dboitnot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

docs/_source/docs/faq.rst Show resolved Hide resolved
docs/_source/docs/faq.rst Outdated Show resolved Hide resolved
docs/_source/docs/faq.rst Show resolved Hide resolved
sceptre/connection_manager.py Show resolved Hide resolved
@jfalkenstein jfalkenstein requested a review from zaro0508 January 17, 2023 16:22
@jfalkenstein
Copy link
Contributor Author

@zaro0508 , I've pushed up your requested changes.

@jfalkenstein jfalkenstein merged commit b88ed55 into Sceptre:master Jan 19, 2023
@jfalkenstein jfalkenstein deleted the jf/making-session-public-property branch January 19, 2023 15:09
jfalkenstein added a commit that referenced this pull request Feb 11, 2023
## 4.0.0 (2023.02.08)

### Added
 - [Resolve #1283] Introducing `sceptre_role`, `cloudformation_service_role` (#1295)
   - These are just iam_role and role_arn renamed to be a lot clearer. See "Deprecations" below.


### Changed
 - [Resolve #1299] Making the ConnectionManager a more "friendly" interface for hooks, resolvers,
   and template handlers (#1287, #1300)
   - This creates adds the public `get_session()` and
     `create_session_environment_variables()` methods to make AWS interactions
     easier and more consistent with individual stack configurations for
     iam_role, profile, and region configurations.
   - The `call()` method now properly distinguishes between default stack
     configurations for profile, region, and `sceptre_role` and setting those to
     `None` to nullify them.
 - Preventing Duplicate Deprecation Warnings from being emitted (#1297)

#### _Potentially_ Breaking Changes
 - The !cmd hook now invokes the passed command using the AWS environment
   variables that correspond with the stack's IAM configurations (i.e. iam_role,
   profile, region). This means that the hook will operate the same as every
   other part of Sceptre and regard how the stack is configured. This should make
   it easier to invoke other tools like AWS CLI with your hooks. However, if
   your project is setting environment variables with the intent to change how
   the command authenticates with AWS (such as a different token, profile, or
   region), these environment variables will be overridden. To maintain the same
   functionality, you should prefix your command with
   `export AWS_SESSION_TOKEN={{environment_variable.AWS_SESSION_TOKEN}} &&` (or
   whatever other environment variable(s) you need to explicitly set).

### Deprecations
 - [Resolve #1283] Deprecating `iam_role`, `role_arn`, and `template_path` (#1295)
   - `iam_role` and `role_arn` have been aliased to `sceptre_role` and
      `cloudformation_service_role`. Using these fields will result in a
      DeprecationWarning.
   - `template_path` has actually been slated for removal since v2.7. `template`
      should be used instead. Using `template_path` will result in a
      DeprecationWarning.
   - All three deprecated StackConfig fields will be removed in v5.0.0.

## 3.3.0 (2023.02.06)

### Added
 - [Resolve #1261] Add coloured differ (#1260)
   - Implements coloured diffs for the diff (difflib) command. Responds to --no-color.
 - [Resolves #1271] Extend stack colourer to include "IMPORT" states (#1272)
 - [Resolves #1179] cloudformation disable-rollback option (#1282)
   - Allow user to disable a cloudformation rollback on a sceptre deployment.

### Changed
 - [Resolve #1098] Deploy docker container to sceptreorg repo (#1265)
   - Deploy sceptre docker images to dockerhub sceptreorg repo instead of cloudreach repo
 - Updating Setuptools and wheel versions to avert security issues
 - [Resolve #1293] Improve the Stack Config Jinja Syntax Error Message to include the Stack Name (#1294)
 - [Resolves #1267] Improve the Stack Config Jinja Error Message to include the Stack Name (#1269)

### Fixed
 - [Resolve #1273] Events start from response time (#1275)
   - Resolves #1273 by starting event filtering from the timestamp returned in
     the AWS response headers rather than relying on the workstation clock.
 - [Resolve #1253] Failed downloads raise error (#1277)
   - Throwing an informative error when the template fails to download instead
     of passing the error message to CloudFormation.
 - [Resolves #1179] Changed disable-rollback default to None (#1288)
   - We want the default value to be None to represent "Do whatever's configured
     in the StackConfig" and True/False will override the StackConfig.

### Nonfunctional
 - Add tweet-release to CircleCI config (#1259)
 - [Resolves #1276] Adopt Black Auto-formatter (#1278)
   - Reformatting all Python files using the Black code formatter. This also
     delivers a new function for generating `__repr__` methods which was needed
     to deal with a line-too-long issue in Template. Per discussion in #1276 this
     PR also disables E203 in flake8.
 - Update sceptre-circleci docker image (#1284)
   - Update to build and test with a docker image that's based on the official
     circleci python docker image.
 - [Resolve #1264] Updating the CDK docs to point to the new sceptre-cdk-handler
   (#1285)
   - This updates our docs to no longer reference the old CDK approach (which
     didn't work with CDK assets). In its place, it references the new
     sceptre-cdk-handler package that covers that functionality.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants