Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Serverless Transform failing when using Intrinsic function within value for "ContentUri" property for "AWS::Serverless::LayerVersion" resource #1159

Closed
razhamma opened this issue Sep 25, 2019 · 11 comments

Comments

@razhamma
Copy link

razhamma commented Sep 25, 2019

Issue Description

The following snippet fails transformation:

Transform: 'AWS::Serverless-2016-10-31'
Resources:
  MyServerlessLayerVersion:
    Type: 'AWS::Serverless::LayerVersion'
    Properties:
      LayerName: !Sub ${AWS::Region}-my-layer
      Description: !Sub ${AWS::Region}-my-layer-description
      ContentUri: !Sub "s3://some-example-bucket-${AWS::Region}/someKey.zip"
      CompatibleRuntimes:
      - python3.7
      RetentionPolicy: Retain

with the Error as: "Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [MyServerlessLayerVersion9b0b8a66e5] is invalid. 'ContentUri' requires Bucket and Key properties to be specified".

Reasoning

  • According to this[1], the "ContentUri" itself is transformed using an S3 util[2] where "ContentUri" is passed to this util as it is(whether a string or an object). However, due to intrinsic function "ContentUri" is treated as an object and then this object is passed to S3 util.
  • Checking into S3 util's code here[3], missing any of the 'Bucket' or 'Key' property of "ContentUri" object results into transformation failure.

Workaround

  • Using the S3 location object format for ContentUri property and using intrinsic functions inside that S3 location object will work as expected without getting any transformation errors.
  • By specifying ContentUri as per S3 location object[4], you won't fail any validation during S3 util parser run and SAM translator will simply pass these values for Key and Bucket property to CloudFormation AWS::Lambda::LayerVersion resource.
  • The correct version for the above mentioned problem facing snippet will look like:
Transform: 'AWS::Serverless-2016-10-31'
Resources:
  MyServerlessLayerVersion:
    Type: 'AWS::Serverless::LayerVersion'
    Properties:
      LayerName: !Sub ${AWS::Region}-my-layer
      Description: !Sub ${AWS::Region}-my-layer-description
      ContentUri: 
        Bucket: !Sub some-example-bucket-${AWS::Region}
        Key: someKey.zip
      CompatibleRuntimes:
      - python3.7
      RetentionPolicy: Retain

Suggestions/Observations

  • CloudFormation resolves intrinsic functions only after transformation is done. So, parsing the string(containing intrinsic function) correctly into 'S3 Location Object' and keeping the exact intrinsic function specifications for this object's properties 'Key' and 'Bucket' during transformation will solve the issue.
  • Another suggestion is making AWS::Serverless Transform Macro capable enough to resolve intrinsic functions beforehand during Transformation. However, it won't be a good approach as we'll be performing same task as CloudFormation is expected to do and it doesn't really go with the sole purpose of just performing SAM based resource translations to CloudFormation specific resources.

Regarding Correct Transformation/Parsing

  • In case if 'ContentUri' string containing intrinsic function was passed to S3 Util paser as a string, it will still fail as S3 util parser itself is using specific python libs(urlparse, parse_qs)[5] and these libs won't be able to parse a string containing CloudFormation specific intrinsic functions.
  • Thus, forcing the string(containing intrinsic function) to be always considered as a string rather than an object and enabling S3 util parser to parse this string by appropriate inclusion/substitution of intrinsic function inside the values for 'Key' and 'Bucket' properties of 'S3 Location Object' can do the job.

References

[1] https://github.com/awslabs/serverless-application-model/blob/master/samtranslator/model/sam_resources.py#L732
[2] https://github.com/awslabs/serverless-application-model/blob/master/samtranslator/model/s3_utils/uri_parser.py
[3] https://github.com/awslabs/serverless-application-model/blob/master/samtranslator/model/s3_utils/uri_parser.py#L61
[4] https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlesslayerversion
[5] https://github.com/awslabs/serverless-application-model/blob/master/samtranslator/model/s3_utils/uri_parser.py#L2

@mingkun2020
Copy link
Contributor

Thanks for reporting the issue, we will be investigating it further. Please watch this channel for more updates, and feel free to reach out.

@hoffa
Copy link
Contributor

hoffa commented Oct 17, 2022

You might be able to get this to work by adding AWS::LanguageExtensions to Transform as such:

Transform:
  - AWS::LanguageExtensions
  - AWS::Serverless-2016-10-31

AWS::LanguageExtensions resolves intrinsic functions if the value is known when Transforms are run.

See #2533 for more information.

@hoffa
Copy link
Contributor

hoffa commented Nov 3, 2022

Closing in favor of #2533.

@hoffa hoffa closed this as completed Nov 3, 2022
@tariromukute
Copy link

You might be able to get this to work by adding AWS::LanguageExtensions to Transform as such:

Transform:
  - AWS::LanguageExtensions
  - AWS::Serverless-2016-10-31

AWS::LanguageExtensions resolves intrinsic functions if the value is known when Transforms are run.

See #2533 for more information.

Thanks @hoffa for the suggestion. I tried the above but didn't manage to get it working. The deployment fails with error. Any ideas?

"FAILED" Status: FAILED. Reason: Transform AWS::Serverless-2016-10-31 failed with: Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [utilitiesLayerfc27b2746a] is invalid. 'ContentUri' is not a valid S3 Uri of the form 's3://bucket/key' with optional versionId query parameter.

@mcrysler-ng
Copy link

Why was this issue closed? I am running into this issue as well so I'm assuming it hasn't been fixed?

@ConnorRobertson
Copy link

@mcrysler-ng have you tried what hoffa mentioned here?
It was closed in favour of this issue. Which provides workarounds and an explanation of why this issue was closed.
Feel free to open a new issue if this problem still persists.

@mcrysler-ng
Copy link

mcrysler-ng commented Nov 22, 2023 via email

@GavinZZ
Copy link
Contributor

GavinZZ commented Nov 22, 2023

@mcrysler-ng would you be able to provide a snippet of your template? Curious why does this workaround not work for you.

ContentUri: 
        Bucket: !Sub some-example-bucket-${AWS::Region}
        Key: someKey.zip

As for the conflict in Language Extension, currently the only known conflict is that you need to depends on ApiGateway Stage resource and you've explicitly used intrinsic function for StageName property. If that doesn't apply to you, please give Language Extension a try too.

@mcrysler-ng
Copy link

mcrysler-ng commented Nov 22, 2023 via email

@mcrysler-ng
Copy link

Snippet included below. The Env parameter is defined earlier in the file and used in almost every resource definition so I know that's not the issue.

This is what is NOT working:

MyLibrary:
    Type: AWS::Serverless::LayerVersion
    Metadata:
      BuildMethod: nodejs18.x
      BuildArchitecture: x86_64
    Properties:
      CompatibleArchitectures:
        - x86_64
      CompatibleRuntimes:
        - nodejs18.x
      ContentUri:
        Bucket: my-bucket
        Key: !Sub "${Env}/${Env}-my-library-name.zip"
      Description: Provides shared library for blah
      LayerName: MyLayer
      LicenseInfo: Available under the MIT-0 license
      RetentionPolicy: Retain

This is what IS working:

MyLibrary:
    Type: AWS::Serverless::LayerVersion
    Metadata:
      BuildMethod: nodejs18.x
      BuildArchitecture: x86_64
    Properties:
      CompatibleArchitectures:
        - x86_64
      CompatibleRuntimes:
        - nodejs18.x
      ContentUri:
        Bucket: my-bucket
        Key: Dev/Dev-my-library-name.zip
      Description: Provides shared library for blah
      LayerName: MyLayer
      LicenseInfo: Available under the MIT-0 license
      RetentionPolicy: Retain

@mcrysler-ng
Copy link

And this is the error I get with the first version above:

Resource handler returned message: "Error occurred while GetObject. S3 Error Code: NoSuchKey. S3 Error Message: The specified key does not exist. (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: b4dd165b-b25c-43d6-945e-0108695b4027; Proxy: null)"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

10 participants