Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add $launchDir and change directory where Nextflow processes are executed #117

Merged
merged 22 commits into from
Sep 3, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
b0f1a8b
Execute Nextflow process in `$launchDir` and add additional variables
lukfor Aug 30, 2023
49efa92
Refactore directory creation and fix remaining tests
lukfor Aug 30, 2023
758a003
Add missing default HOME directory
lukfor Aug 30, 2023
45f6e98
Fix problem with github repositories in script tag
lukfor Aug 30, 2023
3704b51
Merge branch 'main' into fixes/relative-output-dir
lukfor Aug 30, 2023
aadb04f
Cleanup test context
lukfor Aug 31, 2023
cd2a112
Add testcase for `$launchDir`
lukfor Aug 31, 2023
8e3fc42
Add more refactoring
lukfor Aug 31, 2023
42fb4b2
Add variables `$moduleDir` and ``$moduleTestDir`
lukfor Aug 31, 2023
dcd5423
Cleanup params map and fix issues with closures and delegates
lukfor Aug 31, 2023
f77ad8a
Fix issue with empty script and moduleDir
lukfor Aug 31, 2023
f99f330
Remove name property from TestContext to avoid collision
lukfor Aug 31, 2023
53474ea
Add basic documentation
lukfor Aug 31, 2023
94feeb1
Remove auto-sort limitation from docs
lukfor Aug 31, 2023
bb91b2f
Merge branch 'main' into fixes/relative-output-dir
lukfor Sep 2, 2023
de75ac0
Fix merge issues
lukfor Sep 2, 2023
df5a5bc
Fix missing workflow name
lukfor Sep 2, 2023
93134a2
Fix issue with wrong hash in setup method and cleanup dsl1
lukfor Sep 2, 2023
8d2432a
Fix wrong snapshot
lukfor Sep 2, 2023
c8d79ef
Merge branch 'main' into fixes/relative-output-dir
lukfor Sep 2, 2023
2088200
Remove print statements from workflow mock
lukfor Sep 2, 2023
696db5e
Remove println from workflow mock
lukfor Sep 2, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions docs/docs/assertions/assertions.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,11 +38,6 @@ process.out.outputCh = [
]
```

`nf-test` cannot guarantee the order of nextflow channels that contain alternative object types, such as maps. A warning message appears in the console in such cases to highlight the potential issue. To test these channels, the `contains` and `assertContainsInAnyOrder` methods described below can be used
```
Warning: Cannot sort channel, order not deterministic. Unsupported objects types:
```

## Using `with`

This assertions...
Expand Down
85 changes: 31 additions & 54 deletions docs/docs/testcases/global_variables.md
Original file line number Diff line number Diff line change
@@ -1,80 +1,57 @@
# Params and Global Variables
# Global Variables

## Params
The following variables are available and can be used in `setup`, `when`, `then` and `cleanup` closures.

The `params` block is optional and is a simple map that can be used to overwrite Nextflow's input `params`. The `params` block is located in the `when` block of a testcase. You can set params manually:
|Name|Description| Example |
|---|---|---|
|`baseDir` or<br/>`projectDir`| The directory where the `nf-test.config` script is located. | `mypipeline`|
|`moduleDir`| The directory where the module script is located | `mypipeline/modules/mymodule` |
|`moduleTestDir`| The directory where the test script is located | `mypipeline/tests/modules/mymodule` |
|`launchDir`| The directory where the test is run. | `mypipeline/.nf-test/tests/<test_hash>` |
|`metaDir`| The directory where all meta are located (e.g. `mock.nf`).| `mypipeline/.nf-test/tests/<test_hash>/meta` |
|`workDir`| The directory where tasks temporary files are created.| `mypipeline/.nf-test/tests/<test_hash>/work` |
|`outputDir`| An output directory in the `$launchDir` that can be used to store output files. The variable contains the absolute path. If you need a relative outpu directory see [`launchDir` example](#launchdir). | `mypipeline/.nf-test/tests/<test_hash>/output` |
|`params`| Dictionary like object holding all parameters. | |

```Groovy
when {
params {
outdir = "output"
}
}
```

It is also possible to set nested params using the same syntax as in your Nextflow script:
## Examples

```Groovy
when {
params {
output {
dir = "output"
}
}
}
```
### `outputDir`

In addition, you can load the `params` from a JSON file:
This variable points to the directory within the temporary test directory (`.nf-test/tests/<test-dir>/output/`). The variable can be set under params:

```Groovy
when {
params {
load("$baseDir/tests/params.json")
}
params {
outdir = "$outputDir"
}
```

or from a YAML file:
### `baseDir`

This variable points to the directory to locate the base directory of the main nf-test config. The variable can be used e.g. in the process definition to build absolute paths for input files:

```Groovy
when {
params {
load("$baseDir/tests/params.yaml")
}
process {
"""
file1 = file("$baseDir/tests/input/file123.gz")
"""
}
```

nf-test allows to combine both techniques and therefor it is possible to overwrite one or more `params` from the json file:
### `launchDir`

This variable points to the directory where the test is executed. This can be used get access to results that are created in an relative output directory:

```Groovy
when {
params {
load("$baseDir/tests/params.json")
outputDir = "new/output/path"
outdir = "results"
}
}
```

## Global Variables

### `outputDir`

This variable points to the directory within the temporary test directory (`.nf-test/tests/<test-dir>/output/`). The variable can be set under params:

```Groovy
params {
outdir = "$outputDir"
then {
assert path("$launchDir/results").exists()
}
```

### `baseDir`

This variable points to the directory to locate the base directory of the main nf-test config. The variable can be used e.g. in the process definition to build absolute paths for input files:

```Groovy
process {
"""
file1 = file("$baseDir/tests/input/file123.gz")
"""
}
```
```
2 changes: 1 addition & 1 deletion docs/docs/testcases/nextflow_process.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ assert process.out[0].size() == 3

// first element is "hello"
assert process.out[0].get(0) == "hello"
````
```

## Example

Expand Down
64 changes: 64 additions & 0 deletions docs/docs/testcases/params.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Params Dictionary

The `params` block is optional and is a simple map that can be used to overwrite Nextflow's input `params`. The `params` block is located in the `when` block of a testcase. You can set params manually:

```Groovy
when {
params {
outdir = "output"
}
}
```

It is also possible to set nested params using the same syntax as in your Nextflow script:

```Groovy
when {
params {
output {
dir = "output"
}
}
}
```

The `params` map can also be used in the `then` block:

```Groovy
then {
assert params.output == "output"
}
```

## Load params from files

In addition, you can load the `params` from a JSON file:

```Groovy
when {
params {
load("$baseDir/tests/params.json")
}
}
```

or from a YAML file:

```Groovy
when {
params {
load("$baseDir/tests/params.yaml")
}
}
```

nf-test allows to combine both techniques and therefor it is possible to overwrite one or more `params` from the json file:

```Groovy
when {
params {
load("$baseDir/tests/params.json")
outputDir = "new/output/path"
}
}
```
10 changes: 8 additions & 2 deletions example/hello.nf.test
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,14 @@ nextflow_pipeline {

test("hello world example should start 4 processes") {
expect {
assert workflow.success
assert snapshot(workflow).match()
with(workflow) {
assert success
assert trace.tasks().size() == 4
assert "Ciao world!" in stdout
assert "Bonjour world!" in stdout
assert "Hello world!" in stdout
assert "Hola world!" in stdout
}
}
}

Expand Down
5 changes: 3 additions & 2 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,8 @@ nav:
- Workflow Testing: docs/testcases/nextflow_workflow.md
- Process Testing: docs/testcases/nextflow_process.md
- Function Testing: docs/testcases/nextflow_function.md
- Params and Global Variables: docs/testcases/global_variables.md
- Params Dictionary: docs/testcases/params.md
- Global Variables: docs/testcases/global_variables.md
- Running Tests: docs/running-tests.md
- Writing Assertions:
- Power Assertions: docs/assertions/assertions.md
Expand All @@ -51,7 +52,7 @@ nav:
- Regular Expressions: docs/assertions/regular-expressions.md
- FASTA Files: docs/assertions/fasta.md
- Using Third-Party Libraries: docs/assertions/libraries.md
- Command line Interface (CLI):
- Command Line Interface (CLI):
- init: docs/cli/init.md
- generate: docs/cli/generate.md
- test: docs/cli/test.md
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,6 @@ public Integer execute() throws Exception {

String defaultProfile = null;
File defaultConfigFile = null;
File workDir = new File(".nf-test");
String libDir = lib;
boolean defaultWithTrace = true;
try {
Expand All @@ -94,7 +93,6 @@ public Integer execute() throws Exception {
defaultProfile = config.getProfile();
defaultConfigFile = config.getConfigFile();
defaultWithTrace = config.isWithTrace();
workDir = new File(config.getWorkDir());
if (!libDir.isEmpty()) {
libDir += ":";
}
Expand Down Expand Up @@ -151,7 +149,6 @@ public Integer execute() throws Exception {
engine.setScripts(scripts);
engine.setTagQuery(tagQuery);
engine.setDebug(debug);
engine.setWorkDir(workDir);
engine.setUpdateSnapshot(updateSnapshot);
engine.setCleanSnapshot(cleanSnapshot);
engine.setLibDir(libDir);
Expand Down
2 changes: 2 additions & 0 deletions src/main/java/com/askimed/nf/test/config/Config.java
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ public class Config {

public static final String DEFAULT_NEXTFLOW_CONFIG = "tests/nextflow.config";

public static final String DEFAULT_HOME = ".nf-test";

private String workDir = ".nf-test";

private String testsDir = "tests";
Expand Down
Loading