Allow tests to be run concurrently

[petervdonovan]

Maybe this Junit feature can be used in the future to run the target tests concurrently, which would greatly speed them up as they spend large amounts of time waiting on IO.

Originally posted by @oowekyala in #1845 (comment)

In order to do this we would need to update our logging so that it does not just go directly to System.out. As it stands currently, if multiple tests are run at the same time their outputs will all get mixed together.

We would also need to update the test framework so that each federated test uses different network ports. At least for C, the federated tests are maybe what would benefit the most from this because most other test categories except Zephyr tests finish in 3-10 minutes, whereas the federated tests can take 2 hours if many of the tests have deadlocks.

With these changes, I believe that we should be able to start a big thread pool in validateAndRun and submit all the tests in a given category to that thread pool, collecting the output in separate buffers. If we do this then that will motivate us to more rigorously ensure that lfc is not too vulnerable to file system races. It will also worsen our troubles with running out of heap space during tests, but we can see how severe that problem is when we come to it.

I do not consider this issue to be very high priority, but it could lead to a large speedup in running the tests.

[oowekyala]

I have started something in https://github.com/oowekyala/lingua-franca/tree/clem.dynamic-tests

I changed the System.out calls to use the error reporter. Note that #1527 does the same thing in a more elegant way.

My implementation uses JUnit's transparent parallelization capabilities, so we don't have to fiddle ourselves with a thread pool. Here are some of the problems this implementation is facing:

• The Cpp tests fail when run concurrently, because the cmake config, or part of its artifacts Idk, are shared between all tests
• The C federated tests fail for the reason your outline Peter, related to the network ports
• The Rust tests can run concurrently, but show no parallelism. Currently all rust tests are part of a Cargo "workspace", which means they all reuse the same cached compiled dependencies. This avoids redundant work, however, Cargo uses a workspace-wide lock, which effectively kills parallelism when concurrent Cargo tasks are launched. If we removed the workspace, Cargo would start compiling the same dependency crates many times over. Probably the best thing would be to perform code generation for all tests in parallel, then let cargo build all crates in parallel in the workspace. But I don't see how we can do that using the Junit dynamic test feature.

[cmnrd]

The Cpp tests fail when run concurrently, because the cmake config, or part of its artifacts Idk, are shared between all tests.

Indeed, this would be more challenging in C++ as there is shared build logic. However, this can be addressed by running the tests in multiple steps:

1. Run the code generation without invoking the compiler in parallel.
2. Build the C++ code with cmake (this will be parallelized by cmake)
3. Execute all the test binaries.

I don't think extending our test framework is worth the effort though. Parallel builds should be a core functionality of lingo. So perhaps we should run our integration tests through lingo to speed them up.