Query parameters passed into LambdaHandler are missing in backend @RestController

[llax87prog]

To help us debug your issue fill in the basic information below using the options provided

Serverless Java Container version: 1.9.3

Implementations: Spring Boot 2

Framework version: SpringBoot 2.7.11

Frontend service: REST API / API Gateway Proxy Request

Deployment method: SAM

Scenario

We have a Java based lambda micro service deployed and we are seeing intermittently missing some(NOT ALL) of the GET request query parameters for some reason. We have a group of query parameters and some of them are randomly missing in random invocation. Majority of the invocations are fine but maybe in a less than 10% invocation we don't see all the query parameters passed into the backend controllers. The incoming requests are coming from the APIGW. We are using Java-11 and aws-serverless-java-container-springboot2(1.9.3). We are following the same pattern described in this page - https://github.com/awslabs/aws-serverless-java-container/wiki/Quick-start---Spring-Boot2
Note: The issue persist with both Single and Multi Value Query Param
Note: This is intermittent issue and not for all the requests.

Expected behavior

The @RestController should get all the query params sent from API Gateway.

Actual behavior

@RestController not receiving all the query param, some of the the query params missing for few requests.

Steps to reproduce

Create Lambda Handler
public class LambdaHandler implements RequestStreamHandler {
private static final SpringBootLambdaContainerHandler<AwsProxyRequest, AwsProxyResponse> handler;

static {
    try {
        handler = new SpringBootProxyHandlerBuilder<AwsProxyRequest>()
                .defaultProxy()
                .initializationWrapper(new InitializationWrapper())
                .servletApplication()
                .springBootApplication(Application.class)
                .buildAndInitialize();
    } catch (ContainerInitializationException e) {
        throw new RuntimeException("Could not initialize Lambda. ", e);
    }
}

@Override
public void handleRequest(InputStream inputStream, OutputStream outputStream, Context context) throws IOException {
    handler.proxyStream(inputStream, outputStream, context);
}

}

Create @RestController and get the Query Params with @RequestParam Map<String, String> params
{ @GetMapping(path = "/getAll", produces = MediaType.APPLICATION_JSON_VALUE) public ResponseEntity<Object> getAll(final @RequestHeader HttpHeaders headers, final @RequestParam Map<String, String> params) { //Log the Query Param to validate. } }

Full log output

Logs to validate Query Param from AwsProxyRequest and we can see the query params reaching Lambda Handler
"multiValueQueryStringParameters": {
        "localityName": [
            "BLAXLAND"
        ],
        "roadName": [
            "WINNICOOPA"
        ],
        "roadNumber1": [
            "42"
        ],
        "stateTerritoryCode": [
            "NSW"
        ]
    },
    "queryStringParameters": {
        "localityName": "BLAXLAND",
        "roadName": "WINNICOOPA",
        "roadNumber1": "42",
        "stateTerritoryCode": "NSW"
    },

Logs to validate Query Param in @RestController and we can see one of the param missing.
{localityName=[BLAXLAND], stateTerritoryCode=[NSW], roadNumber1=[42]}

Attachment
https://github.com/awslabs/aws-serverless-java-container/assets/94922808/d6bf5164-cf4f-4521-a8c2-964db1a1e9a4
https://github.com/awslabs/aws-serverless-java-container/assets/94922808/8deaa771-89bd-49a7-8a91-206c5a14c089

[valerena]

Hi @llax87prog, this seems like an interesting bug. I tried to reproduce with a similar setup, but I wasn't able to, and I always got all the parameters even after multiple invokes. Is there any more information about your use case that could help us reproduce this?

Maybe:

• Was it consistent if you sent the same parameters that failed before?
• Did you see any sort of pattern in the cases that failed (if you tried with many different parameters)?
• Are you passing any other information (headers, body) other than ?localityName=BLAXLAND&roadName=WINNICOOPA&roadNumber1=42&stateTerritoryCode=NSW

[llax87prog]

Hi @valerena, Thanks for looking into this issue. Yes I tried to find the pattern, but didn't see any pattern which making to drop query param.
May be refer below points if this helps:

1. Make sure you run as Servlet application and not Reactive. Please find the configuration to start your container as Servlet Application.

•    handler = new SpringBootProxyHandlerBuilder<AwsProxyRequest>()
                .defaultProxy()
                .initializationWrapper(new InitializationWrapper())
                **.servletApplication()**
                .springBootApplication(Application.class)
                .buildAndInitialize();
    application.yaml -> 
                    spring:
                       main:
                            **web-application-type: servlet**
  

2. We are seeing this issue when you hit the service with 100 transactions per second through API Gateway for 30 mins or more, few transactions dropping one or two query params randomly.

3. This is not happening to any specific query param.

[llax87prog]

Hi @valerena , Did you had a chance to troubleshoot further on this issue? Kindly let me know, I need a help around to figure out the issue.

[valerena]

Hi @llax87prog . I did more tests and I haven't been able to reproduce the issue. I also thought that if it's too hard to consistently reproduce, it's not too helpful either, so I'm trying to go through the code that does the transformation to see if I can understand why it's failing instead. I'll let you know if I find something that could be causing a problem like this.

And let me know if somehow you can get a consistent way of reproducing.

[valerena]

I think I found where the problem could be coming from: https://github.com/awslabs/aws-serverless-java-container/blob/main/aws-serverless-java-container-core/src/main/java/com/amazonaws/serverless/proxy/internal/servlet/AwsHttpServletRequest.java#L563-L578

The different parameters are read in parallel, but they're all added to the same HashMap output. Since HashMap is not synchronized, then doing these put in multiple threads at the same time can cause problems.

A potential solution should be to call Collections.synchronizedMap() or to use a ConcurrentHashMap (or another alternative to synchronize this or avoid the parallel operations).

I'll check further with the team to confirm this and see what's the best way to solve it. I'll keep you posted.

[llax87prog]

Hi @valerena, Have you got any update further on this issue?

[llax87prog]

@valerena - Can I have update please? If this cannot be resolved, I need to find a alternative approach.

Let me know the update.

[llax87prog]

@valerena - When I can expect new version with your changes? Let me know if you can share beta version to test your changes in my lambda.

[llax87prog]

@valerena - Can I have update please? I would like to deliver my changes to production with this fix.

Let me know if you have beta version to test, So that I can test and confirm once.

[deki]

@llax87prog the proposed change is in https://github.com/valerena/aws-serverless-java-container/tree/change-parameters. Would be great if you can give it a try and provide feedback in #616.

[llax87prog]

@deki , Can you confirm is this fix available for aws-serverless-java-container-springboot3?

I would prefer if you can share the jar, to test this fix. My current code using below dependency.

com.amazonaws.serverless aws-serverless-java-container-springboot3 2.0.0-M2

[deki]

Yes, will let you know once the next release build is available.

[llax87prog]

Hi @deki - thanks for the fix, as provided https://github.com/valerena/aws-serverless-java-container/tree/change-parameters.

I did local build and tried with that, got positive results. The issue mentioned in this did not occur with the fix. I can confirm that fix worked for us.

Total request : 157469

Failure rate without fix : 4%
Failure rate with fix : None

[deki]

Great, thanks for the feedback. We'll get that merged and include it in our next release.