Concurrent Test Execution and maximum thread count

[carlospzurita]

During some stress testing of the validator, related to etf-validator/governance#14, we came upon some issues with concurrent requests. We observed that at a certain number of requests pooled, the system crashes and the test run is stoppped.

Using JMeter, we start a test run with seven tests (test suite Conformance Class 1-7), launching the same request iteratively during 2 minutes.

The exception as it appears in the server log is:

Task java.util.concurrent.FutureTask@4170493d
 rejected from java.util.concurrent.ThreadPoolExecutor@669f091a[Running, pool size = 12, active threads = 12, queued tasks = 12, completed tasks = 111]
    at de.interactive_instruments.etf.webapp.controller.TestRunController.initAndSubmit(TestRunController.java:262)
    at de.interactive_instruments.etf.webapp.controller.TestRunController.start(TestRunController.java:543)

Looking for the source of this issue, we found in the class etf-webapp/src/main/java/de/interactive_instruments/etf/webapp/controller/TestRunController.java, line 108, that the maximum number of threads available is set using the number of cores from the processor

etf-webapp/src/main/java/de/interactive_instruments/etf/webapp/controller/TestRunController.java

Changing manually this value allowed us to pool more requests, up to 100, without any performance issue. ¿Maybe this number could be set in a properties file, or make it dynamically set? This can open up the possibility of elastic resource allocation.

[jonherrmann]

I think it's a good idea to introduce a parameter to control this behaviour.

Since threads in the queue can also block resources, we should also introduce one parameter to explicitly set the maximum size of the queue:

etf.testruns.max.threads
etf.testruns.max.queued

The default for etf.testruns.max.threadsshould be set to to the CPU cores. I would propose a default queue size of three times the CPU cores.

However there is a design flaw in the core: the TestTaskRegistry ctor does not allow to pass a parameter for the queue size. It's always identical to the number of max threads. So in your load test scenario you have increased the parallelism in order to increase the queue size. This might work for small datasets but might lead to issues with bigger ones.

In the next etf-core version 1.1.0 you can call this ctor and set the queue size. Change this line to use the updated library.

[carlospzurita]

Thanks for your comments @jonherrmann, we will try that constructor and keep testing. I suppose that the code of this new version of the etf-core is the one in the branch next, right?

[jonherrmann]

Correct, Carlos. You can either build + install the etf-core locally and use it or just use the library from the ii repository (the snapshot repositories always contain the latest libraries from the next branches) by changing the line to:

    compile group: 'de.interactive_instruments.etf', name: 'etf-core', version: '1.1.0' + project.snapshotSuffix

in the etf-webapp (this has not been changed in the etf-webapp next branch, yet). These are the lines, that have been changed in the etf-core.

[jonherrmann]

Closing this in favour of EIP 48