Making the ConnectionManager a more "friendly" interface for hooks, resolvers, and template handlers

docs/_source/docs/faq.rst

@@ -83,6 +83,61 @@ used.
 
 .. _AWS documentation: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html
 
+.. _using_connection_manager:
+
+How do I call AWS services or use AWS-based tools in my custom hook/resolver/template handler?
+----------------------------------------------------------------------------------------------
+In order to call AWS services in your custom hook/resolver/template handler properly, you should use
+the IAM configurations of the stack where the resolver is being used (unless you need to use a
+different configuration for a specific reason). This means your hook/resolver/handler should honor the
+``profile``, ``region``, and ``iam_role`` configurations as set for your project and/or Stack Config.
+Simply invoking ``boto3.client('s3')`` is _not_ going to regard those and could end up using the
+wrong credentials or not even working.
+
+There is a simple interface available for doing this properly, the
+:py:class:`sceptre.connection_manager.ConnectionManager`. The ConnectionManager is an interface that
+will be pre-configured with each stack's profile, region, and iam_role and will be ready for you to use.
+If you are using an `iam_role`, it will automatically assume that role via STS for making calls to
+AWS so you can just use it the way you want. It is accessible on hooks and resolvers via
+``self.stack.connection_manager`` and on template_handlers via ``self.connection_manager``.
+
+There are three public methods on the ConnectionManager:
+
+- :py:meth:`sceptre.connection_manager.ConnectionManager.call` can be used to directly call a boto3
+  API method on any api service and return the result from boto3. This is perfect when you just need
+  to invoke a boto3 client method without any additional capabilities.
+- :py:meth:`sceptre.connection_manager.ConnectionManager.get_session` can be used to get a boto3 Session
+  object. This is very useful if you need to work with Boto3 Resource objects (like an s3 Bucket) or
+  if you need to create and pass the bot3 session, client, or resource to a third-party framework.
+- :py:meth:`sceptre.connection_manager.ConnectionManager.create_session_environment_variables` creates
+  a dictionary of environment variables used by AWS sdks with all the relevant connection information.
+  This is extremely useful if you are needing to invoke other SDKs using ``subprocess`` and still need
+  the Stack's connection information honored.
+
+
+Using the connection manager, you can use `boto3 <https://boto3.amazonaws.com/v1/documentation/api/latest/index.html>`_
+to perform any AWS actions you need:
+
+.. code-block:: python
+
+   # For example, in your custom resolver:
+   def resolve(self):
+       # You can invoke a lower-level service method like...
+       obj = self.stack.connection_manager.call('s3', 'get_object', {'Bucket': 'my-bucket', 'Key': 'my-key'})
+       # Or you can create higher-level resource objects like...
+       bucket = self.stack.connection_manager.get_session().resource('s3').Bucket('my-bucket')
+       # Or if you need to invoke a third-party tool via a subprocess, you can create the necessary environment
+       # variables like this:
+       environment_variables = self.stack.connection_manager.create_session_environment_variables(
+           include_system_envs=True
+       )
+       list_output = subprocess.run(
+           'aws s3 list-bucket',
+           shell=True,
+           env=environment_variables,
+           capture_output=True
+       ).stdout
+
 
 My CI/CD process uses ``sceptre launch``. How do I delete stacks that aren't needed anymore?
 ---------------------------------------------------------------------------------------------

docs/_source/docs/hooks.rst

@@ -231,3 +231,9 @@ Assume a Sceptre `copy` hook that calls the `cp command`_:
 .. _documentation: http://docs.aws.amazon.com/autoscaling/latest/userguide/as-suspend-resume-processes.html
 .. _this is great place to start: https://docs.python.org/3/distributing/
 .. _cp command: http://man7.org/linux/man-pages/man1/cp.1.html
+
+Calling AWS services in your custom hook
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For details on calling AWS services or invoking AWS-related third party tools in your hooks, see
+:ref:`using_connection_manager`

docs/_source/docs/resolvers.rst

@@ -297,6 +297,12 @@ This resolver can be used in a Stack config file with the following syntax:
    parameters:
      param1: !<custom_resolver_command_name> <value> <optional-aws-profile>
 
+Calling AWS services in your custom resolver
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For details on calling AWS services or invoking AWS-related third party tools in your resolver, see
+:ref:`using_connection_manager`
+
 
 Resolver arguments
 ^^^^^^^^^^^^^^^^^^

docs/_source/docs/template_handlers.rst

@@ -220,3 +220,9 @@ This template handler can be used in a Stack config file with the following synt
 .. _Custom Template Handlers: #custom-template-handlers
 .. _Boto3: https://aws.amazon.com/sdk-for-python/
 .. _http_template_handler key: stack_group_config.html#http-template-handler
+
+Calling AWS services in your custom template_handler
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For details on calling AWS services or invoking AWS-related third party tools in your
+template handler, see :ref:`using_connection_manager`

sceptre/connection_manager.py

@@ -9,16 +9,18 @@
 
 import functools
 import logging
+import os
 import random
 import threading
 import time
-import boto3
+from typing import Optional, Dict
 
-from os import environ
+import boto3
+from botocore.credentials import Credentials
 from botocore.exceptions import ClientError
 
-from sceptre.helpers import mask_key
 from sceptre.exceptions import InvalidAWSCredentialsError, RetryLimitExceededError
+from sceptre.helpers import mask_key
 
 
 def _retry_boto_call(func):
@@ -69,19 +71,21 @@ def decorated(*args, **kwargs):
     return decorated
 
 
+# STACK_DEFAULT is a sentinel value meaning "default to the stack's configuration". This is in
+# contrast with passing None, which would mean "use no value".
+STACK_DEFAULT = "[STACK DEFAULT]"
+
+
 class ConnectionManager(object):
     """
     The Connection Manager is used to create boto3 clients for
     the various AWS services that Sceptre needs to interact with.
 
     :param profile: The AWS credentials profile that should be used.
-    :type profile: str
     :param iam_role: The iam_role that should be assumed in the account.
-    :type iam_role: str
     :param stack_name: The CloudFormation stack name for this connection.
-    :type stack_name: str
     :param region: The region to use.
-    :type region: str
+    :param iam_role_session_duration: The duration to assume the specified iam_role per session.
     """
 
     _session_lock = threading.Lock()
@@ -92,11 +96,14 @@ class ConnectionManager(object):
 
     def __init__(
         self,
-        region,
-        profile=None,
-        stack_name=None,
-        iam_role=None,
-        iam_role_session_duration=None,
+        region: str,
+        profile: Optional[str] = None,
+        stack_name: Optional[str] = None,
+        iam_role: Optional[str] = None,
+        iam_role_session_duration: Optional[int] = None,
+        *,
+        session_class=boto3.Session,
+        get_envs_func=lambda: os.environ,
     ):
 
         self.logger = logging.getLogger(__name__)
@@ -110,6 +117,9 @@ def __init__(
         if stack_name:
             self._stack_keys[stack_name] = (region, profile, iam_role)
 
+        self._session_class = session_class
+        self._get_envs = get_envs_func
+
     def __repr__(self):
         return (
             "sceptre.connection_manager.ConnectionManager(region='{0}', "
@@ -122,27 +132,116 @@ def __repr__(self):
             )
         )
 
-    def _get_session(self, profile, region, iam_role):
+    def get_session(
+        self,
+        profile: Optional[str] = STACK_DEFAULT,
+        region: Optional[str] = STACK_DEFAULT,
+        iam_role: Optional[str] = STACK_DEFAULT,
+    ) -> boto3.Session:
         """
-        Returns a boto session in the target account.
+        Returns a boto3 session for the targeted profile, region, and iam_role.
+
+        For each of profile, region, and iam_role, these values will default to the ConnectionManager's
+        configured default values (which correspond to the Stack's configuration). These values can
+        be overridden, however, by passing them explicitly.
 
-        If a ``profile`` is specified in ConnectionManager's initialiser,
-        then the profile is used to generate temporary credentials to create
-        the Boto session. If ``profile`` is not specified then the default
-        profile is assumed to create the boto session.
+        :param profile: The name of the AWS Profile as configured in the local environment. Passing
+            None will result in no profile being specified. Defaults to the ConnectionManager's
+            configured profile (if there is one).
+        :param region: The AWS Region the session should be configured with. Defaults to the
+            ConnectionManager's configured region.
+        :param iam_role: The IAM role ARN that is assumed using STS to create the session. Passing
+            None will result in no IAM role being assumed. Defaults to the ConnectionManager's
+            configured iam_role (if there is one).
 
         :returns: The Boto3 session.
-        :rtype: boto3.session.Session
         :raises: botocore.exceptions.ClientError
         """
+        profile = self.profile if profile == STACK_DEFAULT else profile
+        region = self.region if region == STACK_DEFAULT else region
+        iam_role = self.iam_role if iam_role == STACK_DEFAULT else iam_role
+        return self._get_session(profile, region, iam_role)
+
+    def create_session_environment_variables(
+        self,
+        profile: Optional[str] = STACK_DEFAULT,
+        region: Optional[str] = STACK_DEFAULT,
+        iam_role: Optional[str] = STACK_DEFAULT,
+        include_system_envs: bool = True,
+    ) -> Dict[str, str]:
+        """Creates the standard AWS environment variables that would need to be passed to a
+        subprocess in a hook, resolver, or template handler and allow that subprocess to work with
+        the currently configured session.
+
+        The environment variables returned by this method should be everything needed for
+        subprocesses to properly interact with AWS using the ConnectionManager's configurations for
+        profile, iam_role, and region. By default, they include the other process environment
+        variables, such as PATH and any others. If you do not want the other environment variables,
+        you can toggle these off via include_system_envs=False.
+
+        | Notes on including system envs:
+        |   * The AWS_DEFAULT_REGION, AWS_REGION, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY
+        |     environment variables (if they are set in the Sceptre process) will be overwritten in
+        |     the returned dict with the correct values from the newly created Session.
+        |   * If the AWS_SESSION_TOKEN environment variable is currently set for the process, this
+        |     will be overwritten with the new session's token (if there is one) or removed from the
+        |     returned environment variables dict (if the new session doesn't have a token).
+
+        :param profile: The name of the AWS Profile as configured in the local environment. Passing
+            None will result in no profile being specified. Defaults to the ConnectionManager's
+            configured profile (if there is one).
+        :param region: The AWS Region the session should be configured with. Defaults to the
+            ConnectionManager's configured region.
+        :param iam_role: The IAM role ARN that is assumed using STS to create the session. Passing
+            None will result in no IAM role being assumed. Defaults to the ConnectionManager's
+            configured iam_role (if there is one).
+        :param include_system_envs: If True, will return a dict with all the system environment
+            variables included. This is useful for creating a complete dict of environment variables
+            to pass to a subprocess. If set to False, this method will ONLY return the relevant AWS
+            environment variables. Defaults to True.
+
+        :returns: A dict of environment variables with the appropriate credentials available for use.
+        """
+        session = self.get_session(profile, region, iam_role)
+        # Set aws environment variables specific to whatever AWS configuration has been set on the
+        # stack's connection manager.
+        credentials: Credentials = session.get_credentials()
+        envs = dict(**self._get_envs()) if include_system_envs else {}
+
+        if include_system_envs:
+            # We don't want a profile specified, since that could interfere with the credentials we're
+            # about to set. Even if we're using a profile, the credentials will already reflect that
+            # profile's configurations.
+            envs.pop("AWS_PROFILE", None)
+
+        envs.update(
+            AWS_ACCESS_KEY_ID=credentials.access_key,
+            AWS_SECRET_ACCESS_KEY=credentials.secret_key,
+            # Most AWS SDKs use AWS_DEFAULT_REGION for the region; some use AWS_REGION
+            AWS_DEFAULT_REGION=session.region_name,
+            AWS_REGION=session.region_name,
+        )
+
+        if credentials.token:
+            envs["AWS_SESSION_TOKEN"] = credentials.token
+        # There might not be a session token, so if there isn't one, make sure it doesn't exist in
+        # the envs being passed to the subprocess
+        elif include_system_envs:
+            envs.pop("AWS_SESSION_TOKEN", None)
+
+        return envs
+
+    def _get_session(
+        self, profile: Optional[str], region: Optional[str], iam_role: Optional[str]
+    ) -> boto3.Session:
         with self._session_lock:
             self.logger.debug("Getting Boto3 session")
             key = (region, profile, iam_role)
 
             if self._boto_sessions.get(key) is None:
                 self.logger.debug("No Boto3 session found, creating one...")
                 self.logger.debug("Using cli credentials...")
-
+                environ = self._get_envs()
                 # Credentials from env take priority over profile
                 config = {
                     "profile_name": profile,
@@ -152,7 +251,7 @@ def _get_session(self, profile, region, iam_role):
                     "aws_session_token": environ.get("AWS_SESSION_TOKEN"),
                 }
 
-                session = boto3.session.Session(**config)
+                session = self._session_class(**config)
                 self._boto_sessions[key] = session
 
                 if session.get_credentials() is None:
@@ -177,7 +276,7 @@ def _get_session(self, profile, region, iam_role):
                     sts_response = sts_client.assume_role(**assume_role_kwargs)
 
                     credentials = sts_response["Credentials"]
-                    session = boto3.session.Session(
+                    session = self._session_class(
                         aws_access_key_id=credentials["AccessKeyId"],
                         aws_secret_access_key=credentials["SecretAccessKey"],
                         aws_session_token=credentials["SessionToken"],

sceptre/hooks/cmd.py

@@ -18,8 +18,9 @@ def run(self):
         :raises: sceptre.exceptions.InvalidTaskArgumentTypeException
         :raises: subprocess.CalledProcessError
         """
+        envs = self.stack.connection_manager.create_session_environment_variables()
         try:
-            subprocess.check_call(self.argument, shell=True)
+            subprocess.check_call(self.argument, shell=True, env=envs)
         except TypeError:
             raise InvalidHookArgumentTypeError(
                 'The argument "{0}" is the wrong type - cmd hooks require '