Karate hangs after finishing the test (with large number of inputs)

[hellomryoo]

Hello,

I am here to report 2 Karate issues (the first one is the original error, and the second one was found when I was trying to replicate the error).

First Issue

I am having an issue with Karate as it hangs after all the test is completed (before the test result gets generated) when there are a lot of inputs (lots of API requests). I tried to mimic the erroneous behavior by only keeping the structure and replace the API calls with the dummy API ('https://www.google.com') and replaced the data with the dummy variable that only works as a counter.

Background information:
The way it works is requestGoogle feature has a Scenario Outline that runs N times (number elements in the JSON array input - totalData.json) and each case would run sendGoogle feature M times (number of elements in the JSON array input - replyToFormData.json). In our case, M is fixed and N is the number of the case I can increase/decrease (for the test case, I set M to be 9). So, the total requests that are being tested is M * N requests.

Issue
Since M is fixed, I can play around with N to see what's causing the issue. Here is what happens:

1. When N=80 with 4GB RAM -> No hanging, works fine (EC2 t2.medium)
2. When N=120 with 4GB RAM -> Hangs (EC2 t2.medium)
3. When N=120 with 8GB RAM -> No hanging, works fine (MacBook Pro 2018 15 inch with 16GB RAM)
4. When N = 180 with 8GB RAM -> Hangs (MacBook Pro 2018 15 inch with 16GB RAM)
5. When N = 180 with 16GB RAM -> No hanging, works fine (MacBook Pro 2018 15 inch with 16GB RAM)
6. When N = 200+ with 16 GB RAM -> Hangs (MacBook Pro 2018 15 inch with 16GB RAM)

On the local machine (MacBook Pro 2018 15 inch with 16GB RAM), I manually assigned the RAM (8/16GB).

I also have changed the Logging from DEBUG to INFO and tried using the parallel execution (with 4). The current project uses parallel execution with 4 threads. Results results = Runner.path("classpath:examples/users/requestGoogle.feature").parallel(4);

Current repo is set to be N = 400, M = 9.

You can use the command mvn clean test to run the project.

Here is the repository for the first issue.
automatedTest.zip

Second Issue

Background information:
While I was duplicating the issue, I wanted to simplify the project as much as possible. So I decided to create a Karate project that has only 1 thing (GET request) and tried to run it as much as I needed in my project. (Around 4,000 + requests). Then this is what happens.

Issue

1. Up to 2700 inputs, it runs and passes.
2. Then I increase to around 3000, it becomes really slow, and a simple GET request to Google fails. (screenshot).
   
3. Then when I increase the input number to like 4000+, the karate won't even start (I think the reading input file with lots of JSON elements is getting stuck while being processed)

When I say the input number, I changed the number of elements in JSON array (totalData.json)

You can use the command mvn clean test to run the project. I used my local machine (MacBook Pro 2018 15 inch with 16GB RAM)

Here is the repository for the second issue.
automatedTest2.zip

[ptrthomas]

@hellomryoo really appreciate the simple example. but I would like you to make one critical change.

please use a karate mock on localhost to be the sample service, not google / external for this case. so we will still simulate the http connections (which may be the bottleneck). if needed you can simulate a response delay. look at the gatling example for reference: https://github.com/intuit/karate/tree/develop/examples/gatling

and it would be nice if you use 0.9.6.RC4 because I don't want to waste time on things that may have been fixed with ongoing code improvements

is it fair to assume that your scenario is non-typical and that you are basically reaching the limits of your machine (CPU + RAM). if this is a blocker, one way you can contribute to the community is to help us make the distributed-testing approach more mature: https://github.com/intuit/karate/wiki/Distributed-Testing

else I am inclined to treat this as very low priority, I have put a help wanted tag on it, and it would require someone to do an interesting profiling exercise. hacktoberfest is coming up and I will add this to the project board.

the dynamic Scenario Outline has a lot of interesting code behind the scenes that snapshots state for each row etc. I'm not at all surprised :) you are welcome to submit a PR - and I would say that is the most useful thing you can do in this situation.

it would also be great if you re-write your test to use a loop, e.g. karate.repeat() and a karate.call(). and in the "called" feature, use a Java class to "generate" the next row of data needed, similar to an Iterator. that would avoid getting the thousands of rows into memory.

let me know your thoughts

[ptrthomas]

moving to project board: https://github.com/intuit/karate/projects/3#card-43330542

[ptrthomas]

hello @hellomryoo are you there. would you be so kind as to take a look at #1394 (comment) and confirm that it looks good. thanks