Release/v1.36.0 (#2022)

* chore: don't install integration tests (#1964) * Remove unnecessary use of comprehension (#1805) * fix: Grammatical error in README.md (#1965) * fix: Added SAR Support Check (#1972) * Added SAR Support Check * Added docstring and Removed Instance Initialization for Class Method * update pyyaml version to get the security update (#1974) * Issue 1508 remove check requiring identity to be required if ReauthorizeEvery equals zero (#1577) * remove check requiring identity to be required Check removed to avoid must specify Identity with at least one of Headers, QueryStrings, StageVariables, or Context. error. This is allowed to be removed from aws console. * set identity to empty dictionary Revert back removal of code section and set identity to empty dictionary instead when function_payload_type is "REQUEST" and no identity defined. * use the correct identity variable fix issue catched by unit test. * Update apigateway.py just set the identity to None * undo change. * remove extra spaces * remove some more spaces * Update test_translator.py remove from test case error_api_invalid_auth as this should be valid. * make the Lambda Authorizer is optional if the authorization caching is not enabled (reference https://docs.aws.amazon.com/apigateway/api-reference/resource/authorizer/#identitySource) * add unit testing to cover the InvalidResourceException in case if the identity values are not exist, and not cached * black reformat Co-authored-by: Mohamed Elasmar <[email protected]> * fix the request parameter parsing, the value can contain dots (#1975) * fix the request parameter parsing, the value can contain dots * fix the unit test for python 2.7 * use built in split, instead of concatenating the string * refactor: Optimize shared API usage plan handling (#1973) * fix: use instance variables for generating shared api usage plan * add extra log statements * fix: Added SAR Support Check (#1972) * Added SAR Support Check * Added docstring and Removed Instance Initialization for Class Method * set log level explicitly * update pyyaml version to get the security update (#1974) * fix: use instance variables for generating shared api usage plan * add extra log statements * set log level explicitly * black formatting * black formatting Co-authored-by: Cosh_ <[email protected]> Co-authored-by: Mohamed Elasmar <[email protected]> * Documentation: fix incorrect header (#1979) Fixed the incorrectly formatted header for HTTP API section * fix: mutable default values in method definitions (#1997) * fix: remove explicit logging level set in single module (#1998) * fix: Crash when using an invalid method in open api (#2001) When customers use auth and define an invalid method in the open api definition, SAM would return a 'server error'. This was actually due to SAM attempting to get the method from the path. If the method was not a supported method and non-lowercase, SAM would attempt to fetch the lower case method and crash with a KeyError. This PR addresses that by checking for the valid methods supported. Co-authored-by: Jacob Fuss <[email protected]> * feat: Resource level attributes support (#2008) * Fix for invalid MQ event source managed policy * Fix for invalid managed policy for MQ, included support for new MQ event source property, updated test cases * Black reformatting * Test case changes * Changed policy name * Modified test cases with new policy name * Added resource attributes and unit tests * Resource attributes initial work * Passthrough attributes for some resources, updated some tests * Resolve merge conflicts * Fixed a typo * Modified implicit api plugin for resource attributes support * Partial update of the tests * Partially updated test cases, black reformatted * Partially updated test templates * Partially updated test templates * Partially updated test templates * Added event bridge support for passthrough resource attributes * Partially updated test templates (up to function with amq kms) * Partially updated test templates (up to sns) * Partially updated test templates (all the ones left) * Prevented passthrough resource attributes from changing layer version hashes * Added test to verify resource passthrough precedence for implicit api * Modified tests related to lambda layer to revert the hash changes, keeping the hash the same with resource attributes added * fix: mutable default values in method definitions (#1997) * fix: remove explicit logging level set in single module (#1998) * run automated tests for resource level attribute support * Skipping metadata in layer hashing * Refactored the classes for TestTranslatorEndToEnd and TestResourceLevelAttributes to share the same parent class * Added new translator tests for version and layer resources * Added new unit tests * Removed after transform resource plugin * Black reformatting * Refactoring implicit api plugin support for DeletionPolicy and UpdateReplacePolicy * Refactoring to improve code quality * Added simple documentation * Black reformatting * Added input template that was missing * Refactoring: use sets instead of lists for implicit api plugin * Changing import to be compatible with py2.7 * Changing test deployment hashes to their actual values Co-authored-by: Mehmet Nuri Deveci <[email protected]> * fix: Fail when Intrinsics are in SourceVPC list instead of IntrinsicsSourceVPC (#1999) * chore: bump version to 1.36.0 (#2014) * Revert "fix: Crash when using an invalid method in open api (#2001)" (#2021) This reverts commit d57b132. Co-authored-by: Chih-Hsuan Yen <[email protected]> Co-authored-by: Harsh Mishra <[email protected]> Co-authored-by: Pranav <[email protected]> Co-authored-by: Cosh_ <[email protected]> Co-authored-by: Mohamed Elasmar <[email protected]> Co-authored-by: daftster <[email protected]> Co-authored-by: Mohamed Elasmar <[email protected]> Co-authored-by: Mehmet Nuri Deveci <[email protected]> Co-authored-by: Ben <[email protected]> Co-authored-by: Jacob Fuss <[email protected]> Co-authored-by: Jacob Fuss <[email protected]>
aws · May 13, 2021 · 0fbcafa · 0fbcafa
1 parent c5d0ed2
commit 0fbcafa
Show file tree

Hide file tree

Showing 72 changed files with 4,108 additions and 1,648 deletions.
diff --git a/README.md b/README.md
@@ -52,7 +52,7 @@ environment that lets you locally build, test, debug, and deploy applications de
 
 ## What is this Github repository? 💻
 
-This GitHub repository contains the SAM Specification, the Python code that translates SAM templates into AWS CloudFormation stacks and lots of examples applications.
+This GitHub repository contains the SAM Specification, the Python code that translates SAM templates into AWS CloudFormation stacks and lots of example applications.
 In the words of SAM developers:
 
 > SAM Translator is the Python code that deploys SAM templates via AWS CloudFormation. Source code is high quality (95% unit test coverage),

diff --git a/docs/internals/generated_resources.rst b/docs/internals/generated_resources.rst
@@ -129,7 +129,7 @@ AWS::Lambda::Permission            MyFunction\ **ThumbnailApi**\ Permission\ **P
   NOTE: ``ServerlessRestApi*`` resources are generated one per stack.
 
 HTTP API
-^^^
+^^^^
 This is called an "Implicit HTTP API". There can be many functions in the template that define these APIs. Behind the 
 scenes, SAM will collect all implicit HTTP APIs from all Functions in the template, generate an OpenApi doc, and create an 
 implicit ``AWS::Serverless::HttpApi`` using this OpenApi. This API defaults to a StageName called "$default" that cannot be

diff --git a/requirements/dev.txt b/requirements/dev.txt
@@ -3,7 +3,7 @@ flake8~=3.8.4
 tox~=3.20.1
 pytest-cov~=2.10.1
 pylint>=1.7.2,<2.0
-pyyaml~=5.3.1
+pyyaml~=5.4
 
 # Test requirements
 pytest~=6.1.1; python_version >= '3.6'

diff --git a/samtranslator/__init__.py b/samtranslator/__init__.py
@@ -1 +1 @@
-__version__ = "1.35.0"
+__version__ = "1.36.0"
diff --git a/samtranslator/intrinsics/resolver.py b/samtranslator/intrinsics/resolver.py
@@ -8,7 +8,7 @@
 
 
 class IntrinsicsResolver(object):
-    def __init__(self, parameters, supported_intrinsics=DEFAULT_SUPPORTED_INTRINSICS):
+    def __init__(self, parameters, supported_intrinsics=None):
         """
         Instantiate the resolver
         :param dict parameters: Map of parameter names to their values
@@ -17,6 +17,8 @@ def __init__(self, parameters, supported_intrinsics=DEFAULT_SUPPORTED_INTRINSICS
         :raises TypeError: If parameters or the supported_intrinsics arguments are invalid
         """
 
+        if supported_intrinsics is None:
+            supported_intrinsics = DEFAULT_SUPPORTED_INTRINSICS
         if parameters is None or not isinstance(parameters, dict):
             raise InvalidDocumentException(
                 [InvalidTemplateException("'Mappings' or 'Parameters' is either null or not a valid dictionary.")]

diff --git a/samtranslator/model/__init__.py b/samtranslator/model/__init__.py
@@ -43,7 +43,11 @@ class Resource(object):
     property_types = None
     _keywords = ["logical_id", "relative_id", "depends_on", "resource_attributes"]
 
-    _supported_resource_attributes = ["DeletionPolicy", "UpdatePolicy", "Condition"]
+    # For attributes in this list, they will be passed into the translated template for the same resource itself.
+    _supported_resource_attributes = ["DeletionPolicy", "UpdatePolicy", "Condition", "UpdateReplacePolicy", "Metadata"]
+    # For attributes in this list, they will be passed into the translated template for the same resource,
+    # as well as all the auto-generated resources that are created from this resource.
+    _pass_through_attributes = ["Condition", "DeletionPolicy", "UpdateReplacePolicy"]
 
     # Runtime attributes that can be qureied resource. They are CloudFormation attributes like ARN, Name etc that
     # will be resolvable at runtime. This map will be implemented by sub-classes to express list of attributes they
@@ -76,6 +80,22 @@ def __init__(self, logical_id, relative_id=None, depends_on=None, attributes=Non
             for attr, value in attributes.items():
                 self.set_resource_attribute(attr, value)
 
+    @classmethod
+    def get_supported_resource_attributes(cls):
+        """
+        A getter method for the supported resource attributes
+        returns: a tuple that contains the name of all supported resource attributes
+        """
+        return tuple(cls._supported_resource_attributes)
+
+    @classmethod
+    def get_pass_through_attributes(cls):
+        """
+        A getter method for the resource attributes to be passed to auto-generated resources
+        returns: a tuple that contains the name of all pass through attributes
+        """
+        return tuple(cls._pass_through_attributes)
+
     @classmethod
     def from_dict(cls, logical_id, resource_dict, relative_id=None, sam_plugins=None):
         """Constructs a Resource object with the given logical id, based on the given resource dict. The resource dict
@@ -318,9 +338,10 @@ def get_passthrough_resource_attributes(self):
 
         :return: Dictionary of resource attributes.
         """
-        attributes = None
-        if "Condition" in self.resource_attributes:
-            attributes = {"Condition": self.resource_attributes["Condition"]}
+        attributes = {}
+        for resource_attribute in self.get_pass_through_attributes():
+            if resource_attribute in self.resource_attributes:
+                attributes[resource_attribute] = self.resource_attributes.get(resource_attribute)
         return attributes
 
 

diff --git a/samtranslator/model/api/api_generator.py b/samtranslator/model/api/api_generator.py
@@ -1,3 +1,4 @@
+import logging
 from collections import namedtuple
 from six import string_types
 from samtranslator.model.intrinsics import ref, fnGetAtt
@@ -24,6 +25,8 @@
 from samtranslator.translator.arn_generator import ArnGenerator
 from samtranslator.model.tags.resource_tagging import get_tag_list
 
+LOG = logging.getLogger(__name__)
+
 _CORS_WILDCARD = "'*'"
 CorsProperties = namedtuple(
     "_CorsProperties", ["AllowMethods", "AllowHeaders", "AllowOrigin", "MaxAge", "AllowCredentials"]
@@ -52,12 +55,20 @@
 GatewayResponseProperties = ["ResponseParameters", "ResponseTemplates", "StatusCode"]
 
 
-class ApiGenerator(object):
-    usage_plan_shared = False
-    stage_keys_shared = list()
-    api_stages_shared = list()
-    depends_on_shared = list()
+class SharedApiUsagePlan(object):
+    """
+    Collects API information from different API resources in the same template,
+    so that these information can be used in the shared usage plan
+    """
+
+    def __init__(self):
+        self.usage_plan_shared = False
+        self.stage_keys_shared = list()
+        self.api_stages_shared = list()
+        self.depends_on_shared = list()
 
+
+class ApiGenerator(object):
     def __init__(
         self,
         logical_id,
@@ -69,6 +80,7 @@ def __init__(
         definition_uri,
         name,
         stage_name,
+        shared_api_usage_plan,
         tags=None,
         endpoint_configuration=None,
         method_settings=None,
@@ -134,6 +146,7 @@ def __init__(
         self.models = models
         self.domain = domain
         self.description = description
+        self.shared_api_usage_plan = shared_api_usage_plan
 
     def _construct_rest_api(self):
         """Constructs and returns the ApiGateway RestApi.
@@ -617,7 +630,11 @@ def _construct_usage_plan(self, rest_api_stage=None):
         # create usage plan for this api only
         elif usage_plan_properties.get("CreateUsagePlan") == "PER_API":
             usage_plan_logical_id = self.logical_id + "UsagePlan"
-            usage_plan = ApiGatewayUsagePlan(logical_id=usage_plan_logical_id, depends_on=[self.logical_id])
+            usage_plan = ApiGatewayUsagePlan(
+                logical_id=usage_plan_logical_id,
+                depends_on=[self.logical_id],
+                attributes=self.passthrough_resource_attributes,
+            )
             api_stages = list()
             api_stage = dict()
             api_stage["ApiId"] = ref(self.logical_id)
@@ -630,18 +647,21 @@ def _construct_usage_plan(self, rest_api_stage=None):
 
         # create a usage plan for all the Apis
         elif create_usage_plan == "SHARED":
+            LOG.info("Creating SHARED usage plan for all the Apis")
             usage_plan_logical_id = "ServerlessUsagePlan"
-            if self.logical_id not in ApiGenerator.depends_on_shared:
-                ApiGenerator.depends_on_shared.append(self.logical_id)
+            if self.logical_id not in self.shared_api_usage_plan.depends_on_shared:
+                self.shared_api_usage_plan.depends_on_shared.append(self.logical_id)
             usage_plan = ApiGatewayUsagePlan(
-                logical_id=usage_plan_logical_id, depends_on=ApiGenerator.depends_on_shared
+                logical_id=usage_plan_logical_id,
+                depends_on=self.shared_api_usage_plan.depends_on_shared,
+                attributes=self.passthrough_resource_attributes,
             )
             api_stage = dict()
             api_stage["ApiId"] = ref(self.logical_id)
             api_stage["Stage"] = ref(rest_api_stage.logical_id)
-            if api_stage not in ApiGenerator.api_stages_shared:
-                ApiGenerator.api_stages_shared.append(api_stage)
-            usage_plan.ApiStages = ApiGenerator.api_stages_shared
+            if api_stage not in self.shared_api_usage_plan.api_stages_shared:
+                self.shared_api_usage_plan.api_stages_shared.append(api_stage)
+            usage_plan.ApiStages = self.shared_api_usage_plan.api_stages_shared
 
             api_key = self._construct_api_key(usage_plan_logical_id, create_usage_plan, rest_api_stage)
             usage_plan_key = self._construct_usage_plan_key(usage_plan_logical_id, create_usage_plan, api_key)
@@ -667,20 +687,30 @@ def _construct_api_key(self, usage_plan_logical_id, create_usage_plan, rest_api_
         """
         if create_usage_plan == "SHARED":
             # create an api key resource for all the apis
+            LOG.info("Creating api key resource for all the Apis from SHARED usage plan")
             api_key_logical_id = "ServerlessApiKey"
-            api_key = ApiGatewayApiKey(logical_id=api_key_logical_id, depends_on=[usage_plan_logical_id])
+            api_key = ApiGatewayApiKey(
+                logical_id=api_key_logical_id,
+                depends_on=[usage_plan_logical_id],
+                attributes=self.passthrough_resource_attributes,
+            )
             api_key.Enabled = True
             stage_key = dict()
             stage_key["RestApiId"] = ref(self.logical_id)
             stage_key["StageName"] = ref(rest_api_stage.logical_id)
-            if stage_key not in ApiGenerator.stage_keys_shared:
-                ApiGenerator.stage_keys_shared.append(stage_key)
-            api_key.StageKeys = ApiGenerator.stage_keys_shared
+            if stage_key not in self.shared_api_usage_plan.stage_keys_shared:
+                self.shared_api_usage_plan.stage_keys_shared.append(stage_key)
+            api_key.StageKeys = self.shared_api_usage_plan.stage_keys_shared
         # for create_usage_plan = "PER_API"
         else:
             # create an api key resource for this api
             api_key_logical_id = self.logical_id + "ApiKey"
-            api_key = ApiGatewayApiKey(logical_id=api_key_logical_id, depends_on=[usage_plan_logical_id])
+            api_key = ApiGatewayApiKey(
+                logical_id=api_key_logical_id,
+                depends_on=[usage_plan_logical_id],
+                attributes=self.passthrough_resource_attributes,
+            )
+            # api_key = ApiGatewayApiKey(logical_id=api_key_logical_id, depends_on=[usage_plan_logical_id])
             api_key.Enabled = True
             stage_keys = list()
             stage_key = dict()
@@ -705,7 +735,12 @@ def _construct_usage_plan_key(self, usage_plan_logical_id, create_usage_plan, ap
             # create a mapping between api key and the usage plan
             usage_plan_key_logical_id = self.logical_id + "UsagePlanKey"
 
-        usage_plan_key = ApiGatewayUsagePlanKey(logical_id=usage_plan_key_logical_id, depends_on=[api_key.logical_id])
+        usage_plan_key = ApiGatewayUsagePlanKey(
+            logical_id=usage_plan_key_logical_id,
+            depends_on=[api_key.logical_id],
+            attributes=self.passthrough_resource_attributes,
+        )
+        # usage_plan_key = ApiGatewayUsagePlanKey(logical_id=usage_plan_key_logical_id, depends_on=[api_key.logical_id])
         usage_plan_key.KeyId = ref(api_key.logical_id)
         usage_plan_key.KeyType = "API_KEY"
         usage_plan_key.UsagePlanId = ref(usage_plan_logical_id)

diff --git a/samtranslator/model/apigateway.py b/samtranslator/model/apigateway.py
@@ -231,8 +231,10 @@ def __init__(
         function_payload_type=None,
         function_invoke_role=None,
         is_aws_iam_authorizer=False,
-        authorization_scopes=[],
+        authorization_scopes=None,
     ):
+        if authorization_scopes is None:
+            authorization_scopes = []
         if function_payload_type not in ApiGatewayAuthorizer._VALID_FUNCTION_PAYLOAD_TYPES:
             raise InvalidResourceException(
                 api_logical_id,
@@ -267,8 +269,9 @@ def _is_missing_identity_source(self, identity):
         query_strings = identity.get("QueryStrings")
         stage_variables = identity.get("StageVariables")
         context = identity.get("Context")
+        ttl = identity.get("ReauthorizeEvery")
 
-        if not headers and not query_strings and not stage_variables and not context:
+        if (ttl is None or int(ttl) > 0) and not headers and not query_strings and not stage_variables and not context:
             return True
 
         return False
@@ -311,7 +314,9 @@ def generate_swagger(self):
                 swagger[APIGATEWAY_AUTHORIZER_KEY]["authorizerCredentials"] = function_invoke_role
 
             if self._get_function_payload_type() == "REQUEST":
-                swagger[APIGATEWAY_AUTHORIZER_KEY]["identitySource"] = self._get_identity_source()
+                identity_source = self._get_identity_source()
+                if identity_source:
+                    swagger[APIGATEWAY_AUTHORIZER_KEY]["identitySource"] = self._get_identity_source()
 
         # Authorizer Validation Expression is only allowed on COGNITO_USER_POOLS and LAMBDA_TOKEN
         is_lambda_token_authorizer = authorizer_type == "LAMBDA" and self._get_function_payload_type() == "TOKEN"

diff --git a/samtranslator/model/eventbridge_utils.py b/samtranslator/model/eventbridge_utils.py
@@ -4,15 +4,15 @@
 
 class EventBridgeRuleUtils:
     @staticmethod
-    def create_dead_letter_queue_with_policy(rule_logical_id, rule_arn, queue_logical_id=None):
+    def create_dead_letter_queue_with_policy(rule_logical_id, rule_arn, queue_logical_id=None, attributes=None):
         resources = []
 
-        queue = SQSQueue(queue_logical_id or rule_logical_id + "Queue")
+        queue = SQSQueue(queue_logical_id or rule_logical_id + "Queue", attributes=attributes)
         dlq_queue_arn = queue.get_runtime_attr("arn")
         dlq_queue_url = queue.get_runtime_attr("queue_url")
 
         # grant necessary permission to Eventbridge Rule resource for sending messages to dead-letter queue
-        policy = SQSQueuePolicy(rule_logical_id + "QueuePolicy")
+        policy = SQSQueuePolicy(rule_logical_id + "QueuePolicy", attributes=attributes)
         policy.PolicyDocument = SQSQueuePolicies.eventbridge_dlq_send_message_resource_based_policy(
             rule_arn, dlq_queue_arn
         )
@@ -41,14 +41,14 @@ def validate_dlq_config(source_logical_id, dead_letter_config):
             raise InvalidEventException(source_logical_id, "No 'Arn' or 'Type' property provided for DeadLetterConfig")
 
     @staticmethod
-    def get_dlq_queue_arn_and_resources(cw_event_source, source_arn):
+    def get_dlq_queue_arn_and_resources(cw_event_source, source_arn, attributes):
         """returns dlq queue arn and dlq_resources, assuming cw_event_source.DeadLetterConfig has been validated"""
         dlq_queue_arn = cw_event_source.DeadLetterConfig.get("Arn")
         if dlq_queue_arn is not None:
             return dlq_queue_arn, []
         queue_logical_id = cw_event_source.DeadLetterConfig.get("QueueLogicalId")
         dlq_resources = EventBridgeRuleUtils.create_dead_letter_queue_with_policy(
-            cw_event_source.logical_id, source_arn, queue_logical_id
+            cw_event_source.logical_id, source_arn, queue_logical_id, attributes
         )
         dlq_queue_arn = dlq_resources[0].get_runtime_attr("arn")
         return dlq_queue_arn, dlq_resources
diff --git a/samtranslator/model/eventsources/cloudwatchlogs.py b/samtranslator/model/eventsources/cloudwatchlogs.py
@@ -43,11 +43,13 @@ def get_source_arn(self):
         )
 
     def get_subscription_filter(self, function, permission):
-        subscription_filter = SubscriptionFilter(self.logical_id, depends_on=[permission.logical_id])
+        subscription_filter = SubscriptionFilter(
+            self.logical_id,
+            depends_on=[permission.logical_id],
+            attributes=function.get_passthrough_resource_attributes(),
+        )
         subscription_filter.LogGroupName = self.LogGroupName
         subscription_filter.FilterPattern = self.FilterPattern
         subscription_filter.DestinationArn = function.get_runtime_attr("arn")
-        if "Condition" in function.resource_attributes:
-            subscription_filter.set_resource_attribute("Condition", function.resource_attributes["Condition"])
 
         return subscription_filter
diff --git a/samtranslator/model/eventsources/pull.py b/samtranslator/model/eventsources/pull.py
@@ -60,7 +60,9 @@ def to_cloudformation(self, **kwargs):
 
         resources = []
 
-        lambda_eventsourcemapping = LambdaEventSourceMapping(self.logical_id)
+        lambda_eventsourcemapping = LambdaEventSourceMapping(
+            self.logical_id, attributes=function.get_passthrough_resource_attributes()
+        )
         resources.append(lambda_eventsourcemapping)
 
         try:
@@ -122,9 +124,6 @@ def to_cloudformation(self, **kwargs):
                     )
             lambda_eventsourcemapping.DestinationConfig = self.DestinationConfig
 
-        if "Condition" in function.resource_attributes:
-            lambda_eventsourcemapping.set_resource_attribute("Condition", function.resource_attributes["Condition"])
-
         if "role" in kwargs:
             self._link_policy(kwargs["role"], destination_config_policy)