Support for ReFrame test fixtures

[vkarak]

We need to support the notion of fixtures for a ReFrame test. A fixture is different from a test dependency and different from a test parameter. The will execute where ReFrame runs and perform operations such as cloning a repo, connecting to a resource that a test needs etc.

Test dependencies in ReFrame differ from fixtures in that they have to deal with the individual test cases (where a test can run and with which programming environments). A fixture will run regardless and will run on the same node that ReFrame itself executes.

Test parameters as discussed in #1567 differ again to fixtures in that the parameterization space is attached to a test we want it to be inherited by its children. Having parameters as fixtures, would require the children to redefine the whole parameter space everytime. We also want parameter attributes to be set.

A typical fixture example, is when you want a test to clone a repo and you don't want to do that for each test case. The current solution is by using test dependencies: have a test to run on a single partition and programming environment and the test doing the actual work depend fully on that one. The problem with this is that you can end up with unresolved dependencies if you run ReFrame with --system=x:part where x:part is not the partition that the test that clones the source is meant to run. This is in fact a restriction that should not exist. You want to clone the repo regardless of where the test executes. That's why we need to introduce the concept of fixtures. Here is an example that shows potential syntax (inspired by pytest fixtures):

import reframe as rfm


@rfm.fixture
def tmp_path():
    yield a_dir_under_stage_prefix()


@rfm.fixture
def clone_repo(tmp_path):
    def _clone_repo(repo):
        return git_clone_repo()

    return clone_repo


class MyTest(rfm.RegressionTest):
    use_fixture('repo', clone_repo('url'))

    def __init__(self):
        self.valid_systems = ['dom:gpu', 'dom:mc']
        self.valid_prog_environs = ['gnu', 'cray']
        self.sourcesdir = self.repo
        ...

Fixtures can be quite intelligent so as not to clone the same repo again and do some reference counting before deleting paths.

In this example, MyTest will clone the test once for all the test cases of MyTest without having to rely on dependencies for this sort of task.

[akesandgren]

What about a completely different test (in another dir) wants to use the same downloaded repo? would this handle that too?
I.e., tests/testA/testA.py uses repo x, tests/testB/testB.py also uses repo x, will repo x stay available once downloaded or will it eventually be cleaned up after A or B has been finished (with all of their dependencies)?

[jjotero]

This is what I had in mind for the fixtures:

A fixture is simply a test that performs a task that is required to run before the main test gets executed.
This differs from the current test dependencies because fixtures have a scope that sets the hierarchy level that a test can run on, also defining which other tests can use its information. This hierarchy cannot be achieved with dependencies, simply because dependencies treat all test equally ignoring the partition and programming environment hierarchies. This is quite obvious even in the simple case of cloning a repo before running a test (as shown in the example below). In such case, the repository should only be cloned once (if needed), regardless of the system, partition, or programming environment used.

From less to more restrictive, the fixture hierarchy levels should be:

• session : Share the test results across the full session. This could be, downloading some files or cloning a git repo. This can only be a run-only test because it must not depend on any programming environment. The test variables valid_systems and valid_prog_environs must be '*'. Here instead of executing every time for every available programming environment and system, only the first one arriving here will execute it, and the subsequent requests will simply use its cached results. This is the main advantage over dependencies.
• partition : Share the test results with the test only for the current partition. This could be fetching the topology of the nodes. This can only be a run-only test because it must not depend on any programming environment. The test variable valid_prog_environs must be '*'. Again, here only the first programming environment will execute the test.
• PrgEnv : Share the test results with the test on the same partition and PrgEnv. This could be a compile-only or any other type of tests.
• test : Create a new private sub-test. This could be cloning a repo and patching some files in it. This must be private, because otherwise this could cause side effects in other tests using that repo.

When using a fixture, other reframe tests can override the fixture's default scope with a more restrictive one if they wish to do so. Changing the scope to a less restrictive one does not make sense. When a fixture is further restricted by a test and the less resticted fixture has already run, we can just deep-copy the fixture and its stagedir.

Some examples:

class CloneRepo(rfm.RunOnlyRegressionTest, scope='session'):
    '''Fixture to clone a repo and share it across the full reframe session.'''
    valid_systems = ['*'] 
    valid_prog_environs = ['*']

    # Undefined variable. Must be defined in order to use this fixture.
    repo = variable(str)

    def __init__(self):
        sourcesdir = self.repo


class CloneReframe(CloneRepo, scope='session'):
    '''Fixture to clone ReFrame.'''
    repo = '[email protected]:eth-cscs/reframe.git'


class CloneAndCheckout(CloneReframe, scope='session'):
    '''Fixture to clone one a repo and checkout a specific version.

    This version would also be shared across the full session. This fixture
    would also do the cloning of the repo, which can lead to repeated cloning
    if some other test has already used the fixture above.
    '''
    ver = variable(str)

    def __init__(self):
        self.postrun_cmds = [f'git checkout {self.ver}']


class Checkout(rfm.RunOnlyRegressionTest, scope='session'):
    '''Fixture that uses the resources of another existing fixture.

    Instead of cloning the resources for every checkout, this fixture uses the
    logic from the fixture `CloneReframe`, and we just copy the sources from that
    fixture's directory.

    Note that this fixture has `session` for its scope. This is because the
    variable `ver` must be set by a derived fixture, and that will make a
    specific version of the code available for the full reframe session.
    '''
    clone = fixture(CloneReframe)
    ver = variable(str)

    valid_systems = ['*']
    valid_prog_environs = ['*']

    def __init__(self):
        sourcesdir = clone.stagedir
        executable = f'git checkout {self.ver}'


class CheckoutV4(Checkout, scope='session'):
    '''Specialised fixture to checkout the version 4.0.'''
    ver = 'v4.0'


@rfm.simple_test
class RemovePipelineFile(rfm.RunOnlyRegressionTest):
    '''Test that removes a file from the sources of a repo (why not).

    This will logically cause a side effect on other tests that might use these
    source files, so the fixture here has to be restricted to the test only.
    Before that, this test wants to remove a file from a specific version,
    so we must specialise the `Checkout` fixture. Now, the checkout fixture has
    a "session" scope, and removing a file from a fixture that is shared
    amonst many test is not a good idea, so we can restrict that fixture to
    act only locally on this test by further restricting the scope directly in
    the fixture built-in.

    This built-in will run the fixture if it hasn't been executed yet.
    Otherwise, this will just return the handle to the test and use the
    existing resources. Though, in this case, we're further restricting the
    scope of the fixture, so the resources of the fixture will be copied from
    its original directory onto the current test's stagedir (because the inline
    scope is now "test").
    '''

    fetch_sources = fixture(CheckoutV4, scope='test')
    def __init__(self):
        the_file = os.path.join(
            fetch_sources.stagedir,
            'reframe/core/pipeline.py'
        )
        executable = f'rm {the_file}'


class CloneMultipleRepos(CloneRepo, scope='session'):
    '''Fixture that clones a few repos.

    Here we reuse the machinery from the `CloneRepo` fixture.
    '''
    my_repo = parameter([
        '[email protected]:eth-cscs/production.git',
        '[email protected]:eth-cscs/reframe.git',
    ])

    def __init__(self):
       self.repo = self.my_repo
       super().__init__()

@rfm.simple_test
class UseFilesFromTwoRepos(rfm.RunOnlyRegressionTest):
    '''Test that requires files from two different repos.

    We use a parametrised fixture to fetch all these sources.
    '''
    # This will return a list with the handles for each instance from the fixture.
    my_repos = fixture(CloneMultipleRepos)

    def __init__(self):
        # Copy the files from the repos into the stagedir.
        repo = my_repos[0]
        self.prerun_cmds = [f'cp {os.path.join(repo.stagedir, file_a.txt)} {self.stagedir}']
        repo = my_repos[1]
        self.prerun_cmds += [f'cp {os.path.join(repo.stagedir, file_b.txt)} {self.stagedir}']

        # ...


@rfm.simple_test
class ForkOffTests(rf.RunOnlyRegressionTest):
    '''For each of the parameters in a fixture, run an independent test.

    The `CloneMultipleRepos` fixture is parametrized with 2 repos. This
    test will run as many times as instances of the parametrized fixture.
    This is the equivalent of parameters, but for fixtures.

    This will generate 2 test, thanks to the `foreach_fixture` built-in.
    '''
    my_repo = foreach_fixture(CloneMultipleRepos)

    def __init__(self):
        self.prerun_cmds = [f'cp {os.path.join(my_repo.stagedir, file_a.txt)} {self.stagedir}']

        # ...

This last example does exactly what is mentioned in #1618.

[vkarak]

We have invested a lot in dependencies and they offer quite some flexibility and nice features, such as concurrent execution, restoring sessions, generating Gitlab pipelines, access to all the information of the target tests. It would be an immense effort to reimplement everything for fixtures. Why don't we look it the other way around? What is missing from the dependencies to act as fixtures? One obvious thing is the ability of a test to "execute anywhere" instead of the current "execute everywhere." Regarding the scopes, I believe that they can be implemented by creating the right dependency graph. Tests now do have access to everything from their parents and if we introduce the concept of "execute anywhere" or "execute with any environment" etc. (aka scope) then we can have what we need.

I like the idea of fixtures being tests or, the other way around, normal tests acting as fixtures, but I would like to see what needs to be added to the tests and/or the dependencies, instead of implementing from scratch a new concept and ending up re-implementing the dependencies.

Practically, if we introduce the concept of a test fixture that is basically a reframe test with the special characteristic of "execute in certain scope", I would like the other tests relate to it through the standard depends_on() function. A new syntax will create confusion between fixtures and dependencies. For me, a fixture, therefore, should be a special test that can execute in any partition or an in any programming environment and not all of them as of now. If another test depends on it, it will simply depend on the instance that has run and then proceed as normal to get anything it needs out of it.

[jjotero]

Where I said above "the main advantage", I should've said "the main difference instead". If the above implied that we should replaced dependencies with fixtures, that's not what I meant :)

I think that by keeping fixtures as test, both dependencies and fixtures could very easily use the same machinery in the background, so I don't think there would be a lot to implement here. Whether we end up supporting fixtures or making dependencies act as fixtures, I think the changes would be the same and only the syntax would differ.

To me, both dependencies and fixtures are very very similar things. However, the scope is the main difference between these two concepts, and also the fact that fixtures execute unconditionally whereas dependencies may or may not execute depending on the when condition. So I guess that the question really is whether we want a "oneAPI" kind of thing for these both concepts, or if instead we split them.

[jjotero]

Here is an update on the fixtures. I've reworked a little bit the syntax from the previous comments and I've now made the examples with the affinity test. This test is ideal for fixtures, because it only needs to be compiled once and then ran multiple times with different options.
In order to maximise reusability, fixtures can use other fixtures; so in this case, the first fixture we should think of is the one that clones the repo with the sources of the affinity tool:

class CloneAffinityRepo(rfm.RunOnlyRegressionTest):
    sourcesdir = 'https://github.com/vkarak/affinity'
    sanity_patterns = sn.assert_found(1)


class PatchAffinityRepo(rfm.RunOnlyRegressionTest):
    '''Use the resources of an existing fixture and apply a patch.

    Here we're altering the files from the original repo. So to avoid
    altering any existing cached clones, we get a fresh clone by
    marking the fixture scope as 'test'.
    '''

    clone = fixture(CloneAffinityRepo, scope='test')

    # ...

    @rfm.run_before('run')
    def apply_patch(self):
        '''The sources can be accessed as self.clone.stagedir'''
        # ...

Just to illustrate the point, we've created a second fixture that applies a patch to the original sources. Fixtures are registered by the fixture or test that uses them, and its scope is also set by the class that uses the fixtures. This is done with the fixture built-in which assigns a handle to the fixture. As shown later, the handle is used to access the fixture and it is needed to allow overriding fixtures and to not hard-code the class name of the fixture in hooks and so on.

So the real deal in this code snipped happens in the PatchAffinityRepo class, where the fixture built-in registers the CloneAffinityRepo as a fixture of the current class. Here, the aim of the PatchAffinityRepo is to modify the original sources of the affinity repo, so we need to have our own private copy to make these changes and prevent side effects in other tests/fixtures that may want the affinity repo sources in their original form. This is achieved by setting the scope of this fixture as 'test', which will force the fixture to execute and not to use any other cached clones that might exist.
Now that we have the sources, we can build the executable:

class BuildAffinityTool(rfm.CompileOnlyRegressionTest):
    '''Use the patched affinity sources and build the executable.

    Here we just read the sources from the `PatchAffinityRepo` fixture
    and build the executable in the current test. Because we're not
    altering the original sources, the scope of the fixture can be set
    to 'session'.
    '''

    src = fixture(PatchAffinityRepo, scope='session')                                                                                   
                                                                                                                                        
    # Build the executable ...                                                                                                          

This BuildAffinityTool fixture uses the patched sources from PatchAffinityRepo and generates the executable. In this case, we assume that this class does not modify/write any files into the PatchAffinityRepo stage, which allows us to set the scope to 'session'. If this class were to modify at all the stage of PatchAffinityRepo the scope should be set to 'test' instead to prevent any side-effects in other tests/fixtures. The next step is to run the executable:

class RunAffinityTestBase(rfm.RunOnlyRegressionTest):                                                                                   
    '''Here we use the `BuildAffinityTool` to generate the executable.

    This one depends on the PrgEnv we're in, so we set that as the
    fixture's scope.
    '''

    aff_tool = fixture(BuildAffinityTool, scope='PrgEnv')

    # .... base run config.


@rfm.simple_test
class RunAffinity1(RunAffinityBase):
    '''First specialisation of the affinity base test.

    This will trigger all the above fixtures before its execution.
    '''

    # ...


@rfm.simple_test
class RunAffinity2(RunAffinityBase):
    '''Another specialisation of the affinity base test.

    This test will reuse the `aff_tool` fixture, which was triggered
    by the `RunAffinity1` test. However, if this test runs on any
    PrgEnv that was not used by `RunAffinity1`, the fixture will run
    again for these new PrgEnvs.
    '''

    # ...

Here the RunAffinityTestBase sets the aff_tool fixture which is inherited by the specialised versions of the test. Note that this fixture contains the executable and thus it depends on the programming environment that was used for its compilation. Hence, we set the scope of this fixture to 'PrgEnv', which ensures that the executable from this fixture will have the matching PrgEnv than the tests that use the executable. With this current setup, we can have many RunOnlyRegressionTests that use the appropriate affinity tool executable, and this executable will only be compiled once per PrgEnv.

Finally, say that someone wants to specialise the build of the BuildAffinityTool class and still reuse the tests that run the affinity tool:

class SpecialBuildAffinityTool(BuildAffinityTool):
    '''This class specialises the `BuildAffinityTool` fixture.

    This could be, for example, adding new compile options.
    '''

    @rfm.run_before('compile')
    def update_compile_opts(self):
       self.build_sytem.cxx_flags += ['-O3']


@rfm.simple_test
class RunAffinity3(RunAffinity2):
    '''Specialisation of the RunAffinity2.

    Here we override the base aff_tool fixture.'''
    aff_tool = fixture(SpecialBuildAffinityTool, scope='PrgEnv')

Because each fixture is assigned a handle in the test class, we can easily override an existing fixture and fully reuse the existing code for the RunAffinity2.

Next in line to all this is what happens when a fixture has one or more parameters. But I think we should agree on the above first 😅.

[vkarak]

I like the interface and particularly, that any test can be used as fixture. You don't have to mark it as such and there is no need for any magic machinery there. There are some things that I'm kind of missing:

1. How does a test get access to the information of its fixtures and what kind of information does it get?
2. I am missing the concept of "setup" and "teardown" which is central to the fixture concept. My guess is that pipeline hooks in the fixture test could take that role. For example, the "setup" could be implied as any "post-init" hook and the "teardown" as any "cleanup" hook in the fixture test.
3. A corollary to (2) is how fixtures will be executed (let aside the scopes), since this would mean that their cleanup phase should be delayed as with the dependencies. And perhaps this has to be done regardless of the "teardown" concept, because even in the "test" scope you don't want to "execute" the fixture until the end, because its cleanup phase will wipe it out.
4. Are the fixtures of the same test going to be reused? Imagine the following example:

class CloneAffinityRepo(rfm.RunOnlyRegressionTest):
    sourcesdir = 'https://github.com/vkarak/affinity'
    sanity_patterns = sn.assert_found(1)


class PatchAffinityRepo(rfm.RunOnlyRegressionTest):

    clone = fixture(CloneAffinityRepo, scope='test')

    @rfm.run_before('run')
    def apply_patch(self):
        '''The sources can be accessed as self.clone.stagedir'''

class BuildAffinityTool(rfm.CompileOnlyRegressionTest):

    src = fixture(PatchAffinityRepo, scope='test')                                                                                   
    clone = fixture(CloneAffinityRepo, scope='test')

The BuildAffinityTool "redefines" a fixture that is already defined somewhere in the hierarchy. (a) Is this valid? (b) If yes, is the clone fixture going to be executed only once? (c) What if the scopes in the "redefinitions" are different?

As a side comment: PrgEnv scope is an ❌ as a name. I would vote for environment or environ.

[jjotero]

Some updates on the progress here. The first draft implementation uses dependencies under the hood. The fixtures are registered by the "host" test during its instantiation and they get picked up later by the loader. From this point onwards, the fixture is a full reframe test, which the host test depends on. However, a pointer to the fixture object will be injected in the host test during its setup stage. This takes me to the points raised by @vkarak above:

1. A fixture is a test attribute. This is injected during the setup pipeline stage. For example:

class MyFixture(rfm.RunOnlyRegressionTest):
    ...

class MyTest(rfm.RunOnlyRegressionTest):
    f = fixture(MyFixture)

    @run_before('compile')
    def print_fixture_stagedir(self):
        print(f'Fixture staged: {self.f.stagedir}')

2. In this case the whole fixture can be seen as the "setup" part and the teardown is the cleanup of the fixture test itself.
3. This is automatically taken care of by the dependencies.
4. The answer to this one is: It depends on the scope. For example, on the case above, both clone fixtures have a 'test' scope, so they will be executed twice. This is because a fixture with a 'test' scope will always execute as a new test. However, if this two clone fixtures use a scope they can share (for example, 'session'), they will only be executed once. In this case, the following statement would be true for the BuildAffinityTool test clone == src.clone.