Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Notebook modules #617

Merged
merged 48 commits into from
Oct 24, 2021
Merged

Notebook modules #617

merged 48 commits into from
Oct 24, 2021

Conversation

grst
Copy link
Member

@grst grst commented Jul 25, 2021

Attempt to implement generic modules for rendering Rmarkdown or jupyter notebooks into HTML reports.
I'm deviating from the one module - one PR rule here, because I feel the two should be discussed together.

The modules support

  • Execute notebook and generate HTML report
  • Jupyter notebooks can be in arbitrary jupytext formats
  • Set sensible defaults for parallel computing (BLAS, MKL, numba etc.)
  • Pass input files (e.g. data tables) to notebooks in a separate channel
  • Retrieve output files ("artifacts") as an output channel
  • Optionally, parametrize notebooks via a separate input channel.

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the module conventions in the contribution docs
  • If necessary, include test data in your PR.
  • Remove all TODO statements.
  • Emit the <SOFTWARE>.version.txt file.
  • Follow the naming conventions.
  • Follow the parameters requirements.
  • Follow the input/output options guidelines.
  • Add a resource label
  • Use BioConda and BioContainers if possible to fulfil software requirements.
  • Ensure that the test works with either Docker / Singularity. Conda CI tests can be quite flaky:
    • PROFILE=docker pytest --tag <MODULE> --symlink --keep-workflow-wd
    • PROFILE=singularity pytest --tag <MODULE> --symlink --keep-workflow-wd
    • PROFILE=conda pytest --tag <MODULE> --symlink --keep-workflow-wd

@grst grst added new module Adding a new module WIP Work in progress labels Jul 25, 2021
@grst grst marked this pull request as ready for review July 31, 2021 13:12
Copy link
Member Author

@grst grst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a few open questions as a code review - would be great if I could get some feedback!

modules/jupyternotebook/main.nf Outdated Show resolved Hide resolved
modules/jupyternotebook/parametrize.nf Outdated Show resolved Hide resolved
modules/rmarkdown/parametrize.nf Outdated Show resolved Hide resolved
modules/jupyternotebook/meta.yml Outdated Show resolved Hide resolved
modules/jupyternotebook/main.nf Outdated Show resolved Hide resolved
tests/modules/jupyternotebook/main.nf Outdated Show resolved Hide resolved
@grst grst requested a review from a team July 31, 2021 13:27
@grst grst requested a review from jianhong July 31, 2021 14:56
@grst
Copy link
Member Author

grst commented Jul 31, 2021

Asking additional review from @jianhong, who showed interest in such a module on slack.

modules/jupyternotebook/main.nf Outdated Show resolved Hide resolved
modules/rmarkdown/main.nf Outdated Show resolved Hide resolved
@grst
Copy link
Member Author

grst commented Aug 2, 2021

I added tests now, see the sister PR nf-core/test-datasets#297 in test-datasets.

grst added a commit to grst/multi-package-containers that referenced this pull request Aug 3, 2021
Copy link
Member

@jfy133 jfy133 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly formatting, one suggestion and one possible suggestion to remove debugging stuff if it's ready.

modules/jupyternotebook/main.nf Outdated Show resolved Hide resolved
modules/jupyternotebook/meta.yml Outdated Show resolved Hide resolved
modules/rmarkdownnotebook/main.nf Outdated Show resolved Hide resolved
modules/rmarkdownnotebook/main.nf Show resolved Hide resolved
tests/config/test_data.config Outdated Show resolved Hide resolved
@grst grst force-pushed the notebook-modules branch from cc4db3d to c1b9234 Compare October 13, 2021 13:17
Copy link
Member

@drpatelh drpatelh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great @grst ! Mostly minor comments. Feel free to merge when they have been addressed!

modules/jupyternotebook/main.nf Outdated Show resolved Hide resolved
modules/jupyternotebook/main.nf Outdated Show resolved Hide resolved
@grst grst merged commit 2ad9816 into nf-core:master Oct 24, 2021
@grst grst deleted the notebook-modules branch October 24, 2021 18:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants