projectDir references become invalid under nf-test

[birnbera]

I've been using the project to great success over the last year (thank you!), so I'm surprised this is the first time I've encountered this issue, however in a pipeline I'm now writing tests for there are numerous references to files in a top-level assets/ directory that contains tool configuration files, etc. In the past, I've been able to make the assumption that the working directory when the pipeline is executed is the one containing main.nf (and assets/). Since nf-test doesn't change the working directory when it stages the test into .nf-test/<uuid>/meta, those paths still work. However, when referring to those paths using the Nextflow $projectDir variable, the test fails because those files don't exist in .nf-test/tests/<uuid>/meta/assets since the new $projectDir from Nextflow's perspective is .nf-test/tests/<uuid>/meta.

My feature request is that there is new field in nf-test.config, e.g. stage: that would specify files and directories besides bin/ and lib/ that should be copied or symlinked (default) to the test run directory. E.g.:

config {
    ...
    stage ['assets', 'some-file.txt', [name: 'a-dir-that-should-be-copied', mode: 'copy']]
}

I do currently have workaround, so this is by no means urgent, but it's annoying to have to remember to add it to all of my test files and seems to rely on a semi-internal detail (i.e. $outputDir/../meta):

import java.nio.file.Files

nextflow_workflow {
    ...

    test("Should run without failures") {

        when {
            // Copy assets directory to the test 'projectDir'
            Files.createSymbolicLink(
                path("${outputDir}/../meta/assets"),
                path("${baseDir}/assets")
            )

            params {
                ...
            }
        }
        then {
            assert workflow.success
        }
    }

[lukfor]

Thanks for reporting this issue!

Your feature request makes definitely sense and is on our todo list!

Best, lukas.

[lukfor]

File staging is implemented in PR #130 and will be available in 0.8.1

[birnbera]

@lukfor Thank you for implementing this very helpful feature! Unfortunately there is still one issue that limits its usefulness. It seems that #117 has now broken relative paths that rely on the current working directory.

For example, I have test params that I load from a file in a when.params block. The params file contains paths to input test files that are relative to the top level of the project. Prior to #117 the current directory never (seemed) to change so relative paths still worked. When I originally made this issue, projectDir pointed elsewhere during tests so it made sense to resolve internal projectDir references by symlinking files to that new location.

The reason I'm leaving the feedback here instead of #117 is that using the new stage directive, you can link files to meta, but the working directory is set to one level up, i.e., meta/../, which still break relative paths. In my mind, it's very surprising that the working directory changes during the test and doesn't seem totally necessary. However, if you choose to stick with that, it would be better to stage files to launchDir rather than meta/. Or give users more options for staging location, but at least by putting the staged files in the new launch/working directory you'd just be replicating the current directory structure.

[MikeWLloyd]

It took me a moment to find this issue, but I had this exact issue as well. We heavily use $projectDir to define paths explicitly throughout our workflows. Using 'stage' fixes my problem, but would it be possible to add more information to the documentation that highlights projectDir staging? It took me awhile to first understand the error and then find this issue.