[feat] Optimise the test case generation

[vkarak]

This PR optimises the generate_testcases() function by eliminating the deep copies completely. This has been one of the major contributing factors in increased load times for large number of tests (at the order of thousands). The execution time overhead of the generate_testcases() has been practically eliminated (from 6.09s to generate 10000 test cases out the HelloTest to 0.1156s on my Mac – 2.3GHz quad-core Intel Core i5). The total gain in listing the checks timed with time, which include (import overheads and print outs) ranges from 25% to 40% on various systems that it was tested. Now the loading phase of tests is fully dominated by the test instantiation:

reframe/reframe/core/decorators.py

Line 67 in 1be1749

def instantiate_all(self, reset_sysenv=0):

This in turn can be broken down in three major factors:

1. The __setattr__ and __getattr__ machinery of the metaclass
2. The dynamic type checking performed for the various test field by the TypedField
3. The deep-copy of the default variable values

These time consuming operations are more difficult to reduce, since especially (1) is at the heart of the test syntax eDSL. For the other two factors, we might think of possible optimisations in the future.

Implementation details

The performance gains in loading the tests come from the following two factors:

1. The deep copies of the test case's partition and environment are completely eliminated. Both the SystemPartition and the Environment are immutable objects, so there is no need to copy them.
2. The deep copy of the test (which is the most expensive) cannot be eliminated, but it was deferred to the setup stage of the pipeline. This way, it gets amortised during the run and it is not paid up front. Also, in order to not pay that cost all at once during the setup stage, the default pipeline timeout was set to 3s, meaning that the framework will only push down the pipeline as many test cases as it can during this window. As these test cases later block waiting for their corresponding build and run jobs, the runtime will advance other tests cases with their setup stage and therefore the deep copy cost of the test is hidden.

Fixes #2497

[victorusu]

lgtm. I think that there is only one English typo... But I am not a native speaker.