Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NPE with AWS Lambda and API Gateway #281

Closed
MounicaRamini opened this issue Sep 23, 2019 · 21 comments
Closed

NPE with AWS Lambda and API Gateway #281

MounicaRamini opened this issue Sep 23, 2019 · 21 comments

Comments

@MounicaRamini
Copy link

MounicaRamini commented Sep 23, 2019

  • Framework version:
  • Implementations: Spring Boot, AWS Lambda and API Gateway

Scenario
NPE when invoking Lambda Function in AWS Lambda Console

Expected behavior
Should be inserting and fetching records from database when Lambda is invoked

Actual Behaviour
Throws below error, when Lambda is executed by using AWS Lambda Test Events

Error Message on AWS Lambda Console

{
  "statusCode": 502,
  "multiValueHeaders": {
    "Content-Type": [
      "application/json"
    ]
  },
  "body": "{\"message\":\"Gateway timeout\"}",
  "base64Encoded": false
}

Full log output on Cloud Wach

%d [%thread] %-5level %logger - %msg%n java.lang.NullPointerException: null
at com.amazonaws.serverless.proxy.internal.servlet.AwsProxyHttpServletRequestReader.readRequest(AwsProxyHttpServletRequestReader.java:48) ~[task/:?]
at com.amazonaws.serverless.proxy.internal.servlet.AwsProxyHttpServletRequestReader.readRequest(AwsProxyHttpServletRequestReader.java:28) ~[task/:?]
at com.amazonaws.serverless.proxy.internal.LambdaContainerHandler.proxy(LambdaContainerHandler.java:174) [task/:?]
at com.deere.lambda.LambdaHandler.LoggingServiceLambdaHandler.handleRequest(LoggingServiceLambdaHandler.java:26) [task/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_201]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_201]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_201]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_201]
at lambdainternal.EventHandlerLoader$PojoMethodRequestHandler.handleRequest(EventHandlerLoader.java:259) [LambdaSandboxJava-1.0.jar:?]
at lambdainternal.EventHandlerLoader$PojoHandlerAsStreamHandler.handleRequest(EventHandlerLoader.java:178) [LambdaSandboxJava-1.0.jar:?]
at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:888) [LambdaSandboxJava-1.0.jar:?]
at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:293) [LambdaSandboxJava-1.0.jar:?]
at lambdainternal.AWSLambda.<clinit>(AWSLambda.java:64) [LambdaSandboxJava-1.0.jar:?]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_201]
at java.lang.Class.forName(Class.java:348) [?:1.8.0_201]
at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:114) [LambdaJavaRTEntry-1.0.jar:?]

LambdaHandler Code

public class LambdaHandler implements RequestHandler<AwsProxyRequest, AwsProxyResponse> {

    private static SpringBootLambdaContainerHandler<AwsProxyRequest, AwsProxyResponse> handler;

    static {
        try {
            handler = SpringBootLambdaContainerHandler.getAwsProxyHandler(SpringBootApplication.class);
        } catch (ContainerInitializationException e) {
            // if we fail here. We re-throw the exception to force another cold start
            e.printStackTrace();
            throw new RuntimeException("Could not initialize Spring Boot Application", e);
        }
    }

    @Override
    public AwsProxyResponse handleRequest(AwsProxyRequest awsProxyRequest, Context context) {
        return handler.proxy(awsProxyRequest, context);
    }
}

Application.class

  public class SpringBootApplication {

        public static void main(String[] args) {

            SpringApplication.run(SpringBootApplication.class, args);
        }

}
@sapessi
Copy link
Collaborator

sapessi commented Sep 23, 2019

Hi @MounicaRamini, can you confirm the version of Serverless Java Container you are using? Looking at that line of code in the latest release it seems harmless since we check for null on multi-value headers in advance.

@MounicaRamini
Copy link
Author

MounicaRamini commented Sep 23, 2019

Hi @sapessi ,
Thanks for replying.

Here is the dependency version

<dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-lambda-java-core</artifactId>
            <version>1.2.0</version>
        </dependency>
 <dependency>
            <groupId>com.amazonaws.serverless</groupId>
            <artifactId>aws-serverless-java-container-spring</artifactId>
            <version>1.3</version>
 </dependency>

@sapessi
Copy link
Collaborator

sapessi commented Sep 23, 2019

Thanks, @MounicaRamini - could you test with the latest release (1.3.2)?

Also as an FYI, in the next release (1.4) we'll break spring boot 2 support into a separate package and include support for reactive apps (webflux) - see #239.

@MounicaRamini
Copy link
Author

MounicaRamini commented Sep 23, 2019

@sapessi
1.3 is the latest version available at Company. So, could you please help me with how to resolve that

@sapessi
Copy link
Collaborator

sapessi commented Sep 23, 2019

I just checked the blame for the null-check line of code. This was indeed a bug fixed in release 1.3.1 (reported in #232). How can we make it easier for you to upgrade and approve version 1.3.2 at your company?

@MounicaRamini
Copy link
Author

Ah!! Am not sure of that. So, the code am following from your repo is good enough to configure and run Lambda Function?

@sapessi
Copy link
Collaborator

sapessi commented Sep 23, 2019

Yes, the code you are using is correct. I'd recommend upgrading to the latest release of the framework. However, for the time being you can just ensure that the test event you send through the Lambda console conforms to the latest contract and includes the multiValueHeaders property. You can find a sample in the API Gateway docs here.

@MounicaRamini
Copy link
Author

@sapessi Thanks for the information.I'll try to test using latest contract format

@MounicaRamini
Copy link
Author

@sapessi
The Lambda function am trying to execute is supposed to make db connection and perform the operations of fetching/inserting records from db.
The Lambda Proxy Integration format am using is also correct as referred to above docs.
So, are there any chances like, if db connection fails and it returns nothing, so we see NullPointerException ? in CloudWatch Logs

@sapessi
Copy link
Collaborator

sapessi commented Sep 23, 2019

I've looked at the code of the 1.3 release. Looks like the NPE is happening when we try to get the elb property from the request. I wonder if it could be caused by the Lambda built-in serialization not reading the annotations in the models. Have you tried using the RequestStreamHandler instead of the RequestHandler? Check the sample here.

@MounicaRamini
Copy link
Author

@sapessi
Yes, I have used RequestStreamHandler interface as well. It still throws the same NPE exception
Here is the LambdaHandler class

public class LambdaHandler implements RequestStreamHandler {
	
	private static SpringBootLambdaContainerHandler<AwsProxyRequest, AwsProxyResponse> handler;
	static {
		try {
			handler = SpringBootLambdaContainerHandler.getAwsProxyHandler(Application.class);
		} catch (ContainerInitializationException e) {
			// if we fail here. We re-throw the exception to force another cold start
			e.printStackTrace();
			throw new RuntimeException("Could not initialize Spring framework", e);
		}
	}
	
	public void handleRequest(InputStream inputStream, OutputStream outputStream, Context context)
			throws IOException {
		handler.proxyStream(inputStream, outputStream, context);
	}

}

Here is the order of Log Output

org.postgresql.util.PSQLException: The connection attempt failed.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:292) ~[task/:?]
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) ~[task/:?]
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195) ~[task/:?]
at org.postgresql.Driver.makeConnection(Driver.java:454) ~[task/:?]
at org.postgresql.Driver.connect(Driver.java:256) ~[task/:?]
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:136) ~[task/:?]
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:369) ~[task/:?]
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:198) ~[task/:?]
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:467) ~[task/:?]
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:541) ~[task/:?]
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115) ~[task/:?]
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112) ~[task/:?]
at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157) ~[task/:?]
..........
java.lang.NullPointerException: null
at com.amazonaws.serverless.proxy.internal.servlet.AwsProxyHttpServletRequestReader.readRequest(AwsProxyHttpServletRequestReader.java:48) ~[task/:?]
at com.amazonaws.serverless.proxy.internal.servlet.AwsProxyHttpServletRequestReader.readRequest(AwsProxyHttpServletRequestReader.java:28) ~[task/:?]
at com.amazonaws.serverless.proxy.internal.LambdaContainerHandler.proxy(LambdaContainerHandler.java:174) [task/:?]
at com.amazonaws.serverless.proxy.internal.LambdaContainerHandler.proxyStream(LambdaContainerHandler.java:209) [task/:?]
at com.deere.lambda.LambdaHandler.LoggingServiceLambdaHandler.handleRequest(LoggingServiceLambdaHandler.java:30) [task/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_201]

Are there any chances like, a NPE is thrown in response because of connection attempt to db is failed ? and it could return nothing and that NPE is not handled ?

If not, would it work with versions below 1.3 ?

@sapessi
Copy link
Collaborator

sapessi commented Sep 24, 2019

it might work with older versions, but I would recommend going to the latest. Can you share the test json you are sending to the function?

@MounicaRamini
Copy link
Author

MounicaRamini commented Sep 24, 2019

Sure @sapessi ,
Here is the json

{
  "resource": "/path/resource",
  "path": "/path/resource",
  "httpMethod": "POST",
  "headers": null,
  "multiValueHeaders": null,
  "queryStringParameters": null,
  "multiValueQueryStringParameters": null,
  "pathParameters": null,
  "stageVariables": null,
  "requestContext": {
    "resourcePath": "/path/resource",
    "httpMethod": "POST",
    "path": "//path/resource",
    "accountId": "accountIdNumber",
    "protocol": "HTTP/1.1",
    "stage": "test-invoke-stage",
    "domainPrefix": "testPrefix",
    "identity": {
      "cognitoIdentityPoolId": null,
      "cognitoIdentityId": null,
      "apiKey": "test-invoke-api-key",
      "principalOrgId": null,
      "cognitoAuthenticationType": null,
      "userArn": "actual arn",
      "apiKeyId": "test-invoke-api-key-id"
    }
  },
  "body": "{ \"Key1\": \"Value1\", \"Key2\": \"Value2\", \"Key3\": \"Vaue3\" }",
  "isBase64Encoded": "false"
}

@sapessi
Copy link
Collaborator

sapessi commented Sep 24, 2019

Out of curiosity, what happens when you send this event (The headers and multiValueHeaders would likely be populated by a client with at least a Content-Type and Accept header):

{
  "resource": "/path/resource",
  "path": "/path/resource",
  "httpMethod": "POST",
  "headers": {},
  "multiValueHeaders": {},
  "queryStringParameters": null,
  "multiValueQueryStringParameters": null,
  "pathParameters": null,
  "stageVariables": null,
  "requestContext": {
    "resourcePath": "/path/resource",
    "httpMethod": "POST",
    "path": "/path/resource",
    "accountId": "accountIdNumber",
    "protocol": "HTTP/1.1",
    "stage": "test-invoke-stage",
    "domainPrefix": "testPrefix",
    "identity": {
      "cognitoIdentityPoolId": null,
      "cognitoIdentityId": null,
      "apiKey": "test-invoke-api-key",
      "principalOrgId": null,
      "cognitoAuthenticationType": null,
      "userArn": "actual arn",
      "apiKeyId": "test-invoke-api-key-id"
    }
  },
  "body": "{ \"Key1\": \"Value1\", \"Key2\": \"Value2\", \"Key3\": \"Vaue3\" }",
  "isBase64Encoded": "false"
}

Also, make sure that the path and httpMethod properties in the event are set to something that actually exists in your Spring application

@sapessi
Copy link
Collaborator

sapessi commented Sep 24, 2019

BTW @MounicaRamini, is the version of the library you are using a private fork of this repo? I can't seem to make sense of the error being at com.amazonaws.serverless.proxy.internal.servlet.AwsProxyHttpServletRequestReader.readRequest(AwsProxyHttpServletRequestReader.java:48) - perhaps I'm looking in the wrong place but I cannot see something that might trigger an NPE at that line with your event.

@MounicaRamini
Copy link
Author

Out of curiosity, what happens when you send this event (The headers and multiValueHeaders would likely be populated by a client with at least a Content-Type and Accept header):

{
  "resource": "/path/resource",
  "path": "/path/resource",
  "httpMethod": "POST",
  "headers": {},
  "multiValueHeaders": {},
  "queryStringParameters": null,
  "multiValueQueryStringParameters": null,
  "pathParameters": null,
  "stageVariables": null,
  "requestContext": {
    "resourcePath": "/path/resource",
    "httpMethod": "POST",
    "path": "/path/resource",
    "accountId": "accountIdNumber",
    "protocol": "HTTP/1.1",
    "stage": "test-invoke-stage",
    "domainPrefix": "testPrefix",
    "identity": {
      "cognitoIdentityPoolId": null,
      "cognitoIdentityId": null,
      "apiKey": "test-invoke-api-key",
      "principalOrgId": null,
      "cognitoAuthenticationType": null,
      "userArn": "actual arn",
      "apiKeyId": "test-invoke-api-key-id"
    }
  },
  "body": "{ \"Key1\": \"Value1\", \"Key2\": \"Value2\", \"Key3\": \"Vaue3\" }",
  "isBase64Encoded": "false"
}

Also, make sure that the path and httpMethod properties in the event are set to something that actually exists in your Spring application

@sapessi Yeah, I have given path and httpMethod values respective to application.So, when I send this event, I get the above errors which I have mentioned. I took this format and values from APIGateway Test and then sent these values from AWS Lambda Console Test Event as API Gateway was timing out

@MounicaRamini
Copy link
Author

BTW @MounicaRamini, is the version of the library you are using a private fork of this repo? I can't seem to make sense of the error being at com.amazonaws.serverless.proxy.internal.servlet.AwsProxyHttpServletRequestReader.readRequest(AwsProxyHttpServletRequestReader.java:48) - perhaps I'm looking in the wrong place but I cannot see something that might trigger an NPE at that line with your event.

@sapessi the version of the library I have picked from my Company's Maven Artifactory

@sapessi
Copy link
Collaborator

sapessi commented Sep 24, 2019

I created a test app with SpringBoot and version 1.3 of the library. I get an NPE in the same file but at a different line (the same of issue #232), the NPE is related to the missing multiValueHeaders and was fixed in release 1.3.1. If this is indeed the issue, simply setting the multiValueHeaders value to {} instead of null in the test event should get rid of this issue. On the timeout:

Timing out is not entirely weird. Cold starts with SpringBoot can be pretty bad. Increase your function's timeout to 30 seconds (API Gateway's limit) and raise the amount of memory allocated to the function (more memory = more CPU and therefore faster start times).

The fact that your exception reports a different line, makes me thing that you are using a forked version of the library. If that's the case, we can't help unless we can test with the same jar you are using.

@MounicaRamini
Copy link
Author

I created a test app with SpringBoot and version 1.3 of the library. I get an NPE in the same file but at a different line (the same of issue #232), the NPE is related to the missing multiValueHeaders and was fixed in release 1.3.1. If this is indeed the issue, simply setting the multiValueHeaders value to {} instead of null in the test event should get rid of this issue. On the timeout:

Timing out is not entirely weird. Cold starts with SpringBoot can be pretty bad. Increase your function's timeout to 30 seconds (API Gateway's limit) and raise the amount of memory allocated to the function (more memory = more CPU and therefore faster start times).

The fact that your exception reports a different line, makes me thing that you are using a forked version of the library. If that's the case, we can't help unless we can test with the same jar you are using.

Setting multiheaders value to { } resolved this issue. Thanks for your help.I really appreciate it 👍
Question related to cold start, I have set my lambda memory_size to 768 MB and timeoout to 120.Should that be changed to 30 to match API Gateway time frame ?

@sapessi
Copy link
Collaborator

sapessi commented Sep 24, 2019

I would, API Gateway will fail the request at 30 seconds anyway so there's no point in keeping Lambda there spinning. One more challenge with this that we are about to resolve in release 1.4: Lambda has a maximum time of 10 seconds for the init step of a function - ie returning from the constructor of your handler class. We have seen Spring applications take longer than 10 seconds (see #210) and Lambda attempting to cold start the function multiple times because of this. In release 1.4, we are introducing an async starter so that we'll take as much as possible of the 10 seconds and then keep the bootstrap process going in the background to avoid killing the function.

@MounicaRamini
Copy link
Author

@sapessi Thanks for the information

sapessi added a commit that referenced this issue Sep 26, 2019
…gets sent an event without the multiValueHeaders property (#281). Also moved the ServletContext to be managed by the ServletRequestReader istead of set on each request by the LambdaContainerHandler. Additional bug fix to set the correct dispatcher type when startAsync is called on the request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants