CompileOnlyRegressionTest and parameter

[yellowhat]

Hi,
I would like to run multiple CompileOnlyRegressionTest depending on a parameter and the RunOnlyRegressionTest depend on them:

import reframe as rfm
import reframe.utility.sanity as sn
from reframe.utility import import_module_from_file, udeps
from reframe.core.decorators import run_after, run_before

@rfm.simple_test
class AppBuildTest(rfm.CompileOnlyRegressionTest):
    valid_systems = ["hpc:local"]
    valid_prog_environs = ["*"]
    build_system = "Spack"
    param_0 = parameter(["app0", "app1"])

    @run_before("compile")
    def setup_build_system(self):
        self.build_system.specs = [self.param_0]
        self.build_system.install_tree = "$SPACK_ROOT/opt/spack"

    @run_before("sanity")
    def set_sanity_patterns(self):
        self.sanity_patterns = sn.assert_found("Updating", self.stdout)

@rfm.simple_test
class AppRunTest(rfm.RunOnlyRegressionTest):
    valid_systems = ["*"]
    valid_prog_environs = ["*"]
    executable = "date"

    @run_after("init")
    def set_depends_on(self):
        self.depends_on("AppBuildTest", udeps.fully)
        # self.depends_on("AppBuildTest_app0", udeps.fully)
        # self.depends_on("AppBuildTest_app1", udeps.fully)

    @sanity_function
    def assert_passed(self):
        return True

This brings to: ./reframe/bin/reframe: could not resolve dependency: ('AppRunTest', 'hpc:local', 'gcc') -> 'AppBuildTest'.
Instead if I have:

        # self.depends_on("AppBuildTest", udeps.fully)
        self.depends_on("AppBuildTest_app0", udeps.fully)
        self.depends_on("AppBuildTest_app1", udeps.fully)

reFrame is able to correctly identify the dependecies.

Am I misusing the parameter funciton?

Thanks

[jjotero]

Hi @yellowhat ! If I understand correctly, you want your RunOnlyRegressionTest to depend in all variants of your CompileOnlyTest, correct? Or do you want to run a single RunOnlyRegressionTest per variant of the CompileOnlyRegressionTests?

If you want the first option, I'm afraid that you are correct and you have to manually set a dependency for each of the variants of your CompileOnlyRegressionTest. Note that you can access a test's parameter space as (in your case) AppBuildTest.param_space. This means that you can generate all the names from the variants of your CompileOnlyRegressionTest and inject all the dependencies by looping through this parameter space. That would save you some time if the param space is rather large :) We are aware that this is not pretty and it is something we're working to fix in the next major release.

In any case, the relationship of these two tests could be better expressed using fixtures (see #1577), which is something that we're also working on right now :)

[yellowhat]

@jjotero Thank for the reply.
Yes I would like the CompileOnlyRegressionTests to depend on all the CompileOnlyTest.

I have tried the following:

import reframe as rfm
import reframe.utility.sanity as sn
from reframe.utility import import_module_from_file, udeps
from reframe.core.decorators import run_after, run_before

@rfm.simple_test
class AppBuildTest(rfm.CompileOnlyRegressionTest):
    valid_systems = ["hpc:local"]
    valid_prog_environs = ["*"]
    build_system = "Spack"
    param_0 = parameter(["app0", "app1"])

    @run_before("compile")
    def setup_build_system(self):
        self.build_system.specs = [self.param_0]
        self.build_system.install_tree = "$SPACK_ROOT/opt/spack"

    @run_before("sanity")
    def set_sanity_patterns(self):
        self.sanity_patterns = sn.assert_found("Updating", self.stdout)

@rfm.simple_test
class AppRunTest(rfm.RunOnlyRegressionTest):
    valid_systems = ["*"]
    valid_prog_environs = ["*"]
    executable = "date"

    @run_after("init")
    def set_depends_on(self):
        for param in AppBuildTest.param_space:
            self.depends_on(f"AppBuildTest_{param[0]}", udeps.fully)
        # self.depends_on("AppBuildTest", udeps.fully)
        # self.depends_on("AppBuildTest_app0", udeps.fully)
        # self.depends_on("AppBuildTest_app1", udeps.fully)

    @sanity_function
    def assert_passed(self):
        return True

But I get the following:

./reframe/bin/reframe: could not resolve dependency: ('AppRunTest', 'generic:default', 'builtin') -> 'AppBuildTest_app0'
./reframe/bin/reframe: skipping all dependent test cases
  - ('AppRunTest', 'generic:default', 'builtin')

[jjotero]

Oh, the param_space is a namespace, so you'll have to write that hook like this

    @run_after("init")
    def set_depends_on(self):
        for v in AppBuildTest.param_space['param_0']:
            self.depends_on(f"AppBuildTest_{v}", udeps.fully)

[yellowhat]

Thanks