Update module template to support container registries #1991

edmundmiller · 2022-11-02T03:36:19Z

PR checklist

This comment contains a description of changes (with reason)
CHANGELOG.md is updated
If you've fixed a bug or added code that should be tested, add tests!
Documentation in docs is updated

Does the prep work for nf-core/modules#2291

github-actions · 2022-11-02T03:36:42Z

Python linting (`black`) is failing

To keep the code consistent with lots of contributors, we run automated code consistency checks.
To fix this CI test, please run:

Install black: pip install black
Fix formatting errors in your pipeline: black .

Once you push these changes the test should pass, and you can hide this comment 👍

We highly recommend setting up Black in your code editor so that this formatting is done automatically on save. Ask about it on Slack for help!

Thanks again for your contribution!

codecov · 2022-11-02T03:42:06Z

Codecov Report

Merging #1991 (984283e) into dev (6bf8f30) will increase coverage by 0.58%.
The diff coverage is 67.18%.

@@            Coverage Diff             @@
##              dev    #1991      +/-   ##
==========================================
+ Coverage   62.01%   62.59%   +0.58%     
==========================================
  Files          42       43       +1     
  Lines        4862     4898      +36     
==========================================
+ Hits         3015     3066      +51     
+ Misses       1847     1832      -15

Impacted Files	Coverage Δ
nf_core/modules/lint/main_nf.py	`65.47% <63.79%> (+0.86%)`	⬆️
nf_core/modules/lint/__init__.py	`83.93% <100.00%> (+0.86%)`	⬆️
nf_core/components/components_utils.py	`32.20% <0.00%> (-22.35%)`	⬇️
nf_core/lint_utils.py	`77.77% <0.00%> (-8.89%)`	⬇️
nf_core/launch.py	`63.45% <0.00%> (-0.24%)`	⬇️
nf_core/components/components_install.py	`69.64% <0.00%> (ø)`
nf_core/modules/modules_repo.py	`79.51% <0.00%> (+1.29%)`	⬆️
nf_core/subworkflows/install.py	`22.50% <0.00%> (+7.68%)`	⬆️
... and 1 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

mahesh-panchal

Seems the JSON schema is also missing the new param.

nf_core/modules/bump_versions.py

drpatelh · 2022-11-02T10:12:13Z

Think we need to iron out how we get nf-core download working with these new changes before this is merged. See Slack discussion here and feature request nextflow-io/nextflow#3340

This isn't testing the creation of a new module, it's just checking that linting works in a new modules _repo_

This isn't as pretty as the biocontainers api for now, but it works and was recommended in the biocontainers gitter and is how galaxy pulls the images in. Useful links from the gitter: https://matrix.to/#/!wNQPhGTQvTINsnORJY:gitter.im/$8j7bYssM3ZqiMPLnqPQa5BMKjJppeQ_59H-5LVXkYo4?via=gitter.im&via=matrix.org https://github.com/galaxyproject/galaxy/blob/6da2f0e98cfd99b33b8fdb8d63f36096f477aa73/lib/galaxy/tool_util/deps/mulled/util.py#L225 https://github.com/galaxyproject/galaxy/blob/6da2f0e98cfd99b33b8fdb8d63f36096f477aa73/lib/galaxy/tool_util/deps/mulled/util.py#L18 https://github.com/common-workflow-language/cwltool/blob/main/cwltool/software_requirements.py

Co-authored-by: mahesh-panchal <[email protected]>

nf_core/modules/lint/main_nf.py

tests/test_modules.py

mirpedrol · 2022-11-03T09:01:35Z

nf_core/utils.py

-                        singularity_image = all_singularity[k]["image"]
-                        current_date = date
-                return docker_image["image_name"], singularity_image["image_name"]
+                return response.json()["tags"][0]["name"]


I tried with hla-la and I get an older version than for bioconda. Should we check if versions match and downgrade bioconda if they don't?

conda (params.enable_conda ? "bioconda::hla-la=1.0.3" : null) def container_image = "hla-la:1.0.1--h1b026d1_5" container [ params.container_registry ?: 'quay.io/biocontainers' , container_image ].join('/')

Can you manually check if there's a matching hla-la version? I was just lazy and hoped asking it for only one image gave you the latest 😬

I can check that myself.

Okay there is a 1.0.3 version, the problem is it got pushed 4 months ago, but the 1.0.1 version with the _5 build got pushed 3 months ago.

IDK what that's about, or which one we'd want to choose in that case.
https://quay.io/repository/biocontainers/hla-la?tab=tags

Oh, I see that's an edge case. Then I guess we can let the user decide, but maybe still print a warning if the versions are different.

tests/modules/lint.py

Co-authored-by: Júlia Mir Pedrol <[email protected]>

mashehu · 2024-01-09T07:50:46Z

I think this is now implemented, please reopen @edmundmiller if that is not the case

feat(module-template): Introduce new container registry format

c7f7920

edmundmiller self-assigned this Nov 2, 2022

edmundmiller force-pushed the container_registry branch from d67fea7 to a16467e Compare November 2, 2022 03:42

mahesh-panchal reviewed Nov 2, 2022

View reviewed changes

nf_core/modules/bump_versions.py Outdated Show resolved Hide resolved

edmundmiller and others added 17 commits November 2, 2022 08:48

feat(pipeline-template): Add container_registry support

76efdd4

feat(lint): Add container_registry to nextflow.config checks

562714c

refactor(modules/lint): Create seperate check_container_section

3979e27

test(modules): Update name of test

63e86a3

This isn't testing the creation of a new module, it's just checking that linting works in a new modules _repo_

test(modules): Add lint step to creation of new module

ba5ba58

fix(#1979): Add local modules to single linting

7df4e91

test: Fresh module should have no warnings

4b48f7b

feat(modules): Add container to deprecated syntax

359c434

test(biocontainers): Use a mock

5ccce4f

test: Add an util to check if a PR has been merged

2bd465b

fix(modules/lint): Remove container versions do not match

da1dc33

refactor: Join singularity and docker into container tag

d9b3f8f

refactor: Move bioconda_package into it's own function

47d6ef7

test: Add test for _check_bioconda_package

e638bc5

chore: Update CHANGELOG

0fc7cb1

style: Fix typo

799682b

Co-authored-by: mahesh-panchal <[email protected]>

edmundmiller force-pushed the container_registry branch from a16467e to 799682b Compare November 2, 2022 13:50

mirpedrol reviewed Nov 3, 2022

View reviewed changes

Apply suggestions from code review

984283e

Co-authored-by: Júlia Mir Pedrol <[email protected]>

mashehu closed this Jan 9, 2024

muffato deleted the container_registry branch September 7, 2024 22:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update module template to support container registries #1991

Update module template to support container registries #1991

edmundmiller commented Nov 2, 2022 •

edited

Loading

github-actions bot commented Nov 2, 2022

codecov bot commented Nov 2, 2022 •

edited

Loading

mahesh-panchal left a comment

drpatelh commented Nov 2, 2022

mirpedrol Nov 3, 2022

edmundmiller Nov 3, 2022

edmundmiller Nov 3, 2022 •

edited

Loading

mirpedrol Nov 4, 2022

mashehu commented Jan 9, 2024

Update module template to support container registries #1991

Update module template to support container registries #1991

Conversation

edmundmiller commented Nov 2, 2022 • edited Loading

PR checklist

github-actions bot commented Nov 2, 2022

Python linting (black) is failing

codecov bot commented Nov 2, 2022 • edited Loading

Codecov Report

mahesh-panchal left a comment

Choose a reason for hiding this comment

drpatelh commented Nov 2, 2022

mirpedrol Nov 3, 2022

Choose a reason for hiding this comment

edmundmiller Nov 3, 2022

Choose a reason for hiding this comment

edmundmiller Nov 3, 2022 • edited Loading

Choose a reason for hiding this comment

mirpedrol Nov 4, 2022

Choose a reason for hiding this comment

mashehu commented Jan 9, 2024

edmundmiller commented Nov 2, 2022 •

edited

Loading

Python linting (`black`) is failing

codecov bot commented Nov 2, 2022 •

edited

Loading

edmundmiller Nov 3, 2022 •

edited

Loading