Bug: sam local invoke still print the response in latin-1

[guanlinz]

Description:

Passing the event in Chineses to sam local invoke produces results like unsupported character.

This issue only hapeens in STDOUT with sam local invoke, I can get correct and human readable Chineses character via API Gateway

I go through the closed issue and got this: #2494, seems like the issue is fixed in 2021, but in my testing result, it didn't.

Steps to reproduce:

SAM Template:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  helloword-app

  Sample SAM Template for helloword-app

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3
    MemorySize: 128

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.9
      Architectures:
        - x86_64
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get

Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  HelloWorldApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
  HelloWorldFunction:
    Description: "Hello World Lambda Function ARN"
    Value: !GetAtt HelloWorldFunction.Arn
  HelloWorldFunctionIamRole:
    Description: "Implicit IAM Role created for Hello World function"
    Value: !GetAtt HelloWorldFunctionRole.Arn

Event.json:

{
  "Records": [
    {
      "body": "你好北京"
    }
  ]
}

Lambda code:

import json

def lambda_handler(event, context):
    response = {  
        "statusCode": 200,  
        "body": json.dumps(event, ensure_ascii=False)
    }  

    return response

Invoking method:
sam local invoke -e events/event.json

Observed result:

STDOUT with sam local invoke -e events/event.json :

[ec2-user@ip-172-31-25-214 helloword-app]$  sam local invoke -e events/event.json 
Invoking app.lambda_handler (python3.9)                                                                                                                                                                                 
Local image is up-to-date                                                                                                                                                                                               
Using local image: public.ecr.aws/lambda/python:3.9-rapid-x86_64.                                                                                                                                                       
                                                                                                                                                                                                                        
Mounting /home/ec2-user/sam-workshop/helloword-app/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime container                                                                                
START RequestId: 4fc26e4f-175b-4818-9279-d09ba190a184 Version: $LATEST
{"statusCode": 200, "body": "{\"Records\": [{\"body\": \"\u4f60\u597d\u5317\u4eac\"}]}"}END RequestId: 4fc26e4f-175b-4818-9279-d09ba190a184

Expected result:

It should print the meanful Chineses character in STDOUT, like in below

REPORT RequestId: 048587d1-8017-44a9-8887-64b6bdd0c84f  Init Duration: 0.12 ms  Duration: 60.01 ms      Billed Duration: 61 ms  Memory Size: 128 MB     Max Memory Used: 128 MB
[ec2-user@ip-172-31-25-214 helloword-app]$  sam local invoke -e events/event.json 
Invoking app.lambda_handler (python3.9)                                                                                                                                                                                 
Local image is up-to-date                                                                                                                                                                                               
Using local image: public.ecr.aws/lambda/python:3.9-rapid-x86_64.                                                                                                                                                       
                                                                                                                                                                                                                        
Mounting /home/ec2-user/sam-workshop/helloword-app/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime container                                                                                
START RequestId: 4fc26e4f-175b-4818-9279-d09ba190a184 Version: $LATEST
{"statusCode": 200, "body": "{\"Records\": [{\"body\": \"你好北京\"}"}END RequestId: 4fc26e4f-175b-4818-9279-d09ba190a184
REPORT RequestId: 4fc26e4f-175b-4818-9279-d09ba190a184  Init Duration: 0.04 ms  Duration: 60.12 ms      Billed Duration: 61 ms  Memory Size: 128 MB     Max Memory Used: 128 MB

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Amazon Linux 2
2. sam --version: SAM CLI, version 1.84.0
3. AWS region: cn-north-1

# Paste the output of `sam --info` here

[ec2-user@ip-172-31-25-214 helloword-app]$ sam --info
{
  "version": "1.84.0",
  "system": {
    "python": "3.11.3",
    "os": "Linux-5.10.130-118.517.amzn2.x86_64-x86_64-with-glibc2.26"
  },
  "additional_dependencies": {
    "docker_engine": "20.10.17",
    "aws_cdk": "2.58.1 (build 3d8df57)",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

[jfuss]

@guanlinz Thanks for the report. I started a PR to hopefully address this.

The issue you linked was about the event to the "local" Lambda not being encoded not the output.

What do you mean by "This issue only hapeens in STDOUT with sam local invoke, I can get correct and human readable Chineses character via API Gateway"? I need to validate in Lambda but it's possible we are matching Lambda behavior and sam local invoke is not the same as API Gateway.

Does this work in sam local start-api?

[jfuss]

What do you mean by "This issue only hapeens in STDOUT with sam local invoke, I can get correct and human readable Chineses character via API Gateway"? I need to validate in Lambda but it's possible we are matching Lambda behavior and sam local invoke is not the same as API Gateway.
Does this work in sam local start-api?

Just tested on the AWS Lambda console. The PR I submitted now matches the same response. So likely we just were not encoding things correctly locally for some time.

[guanlinz]

@guanlinz Thanks for the report. I started a PR to hopefully address this.

The issue you linked was about the event to the "local" Lambda not being encoded not the output.

What do you mean by "This issue only hapeens in STDOUT with sam local invoke, I can get correct and human readable Chineses character via API Gateway"? I need to validate in Lambda but it's possible we are matching Lambda behavior and sam local invoke is not the same as API Gateway.

Does this work in sam local start-api?

yes, it works.

Here are the detail:

Start the local api:

[ec2-user@ip-172-31-25-214 helloword-app]$ sam local start-api
Initializing the lambda functions containers.                                                                                                                                                            
Local image is up-to-date                                                                                                                                                                                
Using local image: public.ecr.aws/lambda/python:3.9-rapid-x86_64.                                                                                                                                        
                                                                                                                                                                                                         
Mounting /home/ec2-user/sam-workshop/helloword-app/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime container                                                                 
Containers Initialization is done.                                                                                                                                                                       
Mounting HelloWorldFunction at http://127.0.0.1:3000/hello [GET]                                                                                                                                         
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. If you 
used sam build before running local commands, you will need to re-run sam build for the changes to be picked up. You only need to restart SAM CLI if you update your AWS SAM template                    
2023-06-01 01:46:09 WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:3000
2023-06-01 01:46:09 Press CTRL+C to quit
Invoking app.lambda_handler (python3.9)                                                                                                                                                                  
Reuse the created warm container for Lambda function 'HelloWorldFunction'                                                                                                                                
Lambda function 'HelloWorldFunction' is already running                                                                                                                                                  
START RequestId: 763ffdf3-e7f0-4c30-88bd-3ce009289975 Version: $LATEST
END RequestId: 763ffdf3-e7f0-4c30-88bd-3ce009289975
REPORT RequestId: 763ffdf3-e7f0-4c30-88bd-3ce009289975  Init Duration: 0.13 ms  Duration: 60.32 ms      Billed Duration: 61 ms  Memory Size: 128 MB     Max Memory Used: 128 MB
No Content-Type given. Defaulting to 'application/json'.                                                                                                                                                 
2023-06-01 01:48:03 127.0.0.1 - - [01/Jun/2023 01:48:03] "GET /hello HTTP/1.1" 200 -
2023-06-01 01:48:03 127.0.0.1 - - [01/Jun/2023 01:48:03] "GET /favicon.ico HTTP/1.1" 403 -

Request:

curl --location --request GET 'http://127.0.0.1:3000/hello' \
--header 'Content-Type: application/json' \
--data '{
    "北京": "你好"
}'

Response:

{
    "body": "{\r\n    \"北京\": \"你好\"\r\n}",
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate, br",
        "Cache-Control": "no-cache",
        "Connection": "keep-alive",
        "Content-Length": "28",
        "Content-Type": "application/json",
        "Host": "127.0.0.1:3000",
        "Postman-Token": "f4cd5290-b8cc-4517-86f0-71f81f497e4f",
        "User-Agent": "PostmanRuntime/7.32.2",
        "X-Forwarded-Port": "3000",
        "X-Forwarded-Proto": "http"
    },
    "httpMethod": "GET",
    "isBase64Encoded": false,
    "multiValueHeaders": {
        "Accept": [
            "*/*"
        ],
        "Accept-Encoding": [
            "gzip, deflate, br"
        ],
        "Cache-Control": [
            "no-cache"
        ],
        "Connection": [
            "keep-alive"
        ],
        "Content-Length": [
            "28"
        ],
        "Content-Type": [
            "application/json"
        ],
        "Host": [
            "127.0.0.1:3000"
        ],
        "Postman-Token": [
            "f4cd5290-b8cc-4517-86f0-71f81f497e4f"
        ],
        "User-Agent": [
            "PostmanRuntime/7.32.2"
        ],
        "X-Forwarded-Port": [
            "3000"
        ],
        "X-Forwarded-Proto": [
            "http"
        ]
    },
    "multiValueQueryStringParameters": null,
    "path": "/hello",
    "pathParameters": null,
    "queryStringParameters": null,
    "requestContext": {
        "accountId": "123456789012",
        "apiId": "1234567890",
        "domainName": "127.0.0.1:3000",
        "extendedRequestId": null,
        "httpMethod": "GET",
        "identity": {
            "accountId": null,
            "apiKey": null,
            "caller": null,
            "cognitoAuthenticationProvider": null,
            "cognitoAuthenticationType": null,
            "cognitoIdentityPoolId": null,
            "sourceIp": "127.0.0.1",
            "user": null,
            "userAgent": "Custom User Agent String",
            "userArn": null
        },
        "path": "/hello",
        "protocol": "HTTP/1.1",
        "requestId": "6f608f14-1e96-48bd-aa07-79d846cecb8c",
        "requestTime": "01/Jun/2023:01:46:03 +0000",
        "requestTimeEpoch": 1685583963,
        "resourceId": "123456",
        "resourcePath": "/hello",
        "stage": "Prod"
    },
    "resource": "/hello",
    "stageVariables": null,
    "version": "1.0"
}

[github-actions]

Patch is released in v1.95.0. Closing