Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

List process container images in preview mode #4069

Merged
merged 71 commits into from
Aug 24, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
cb555cb
Dump container image for each process during preview run
bentsherman Jun 30, 2023
e1640fb
Support any directives, save report to JSON file
bentsherman Jul 5, 2023
a84511a
Add -preview-report CLI option to enable preview report
bentsherman Jul 6, 2023
2a85e7a
Merge branch 'master' into 3340-preview-container-images
bentsherman Jul 27, 2023
181c8b3
Apply suggestions from review
bentsherman Jul 27, 2023
aef325e
Merge branch 'master' into 3340-preview-container-images
pditommaso Jul 30, 2023
b330461
Merge branch '3340-preview-container-images' of github.com:nextflow-i…
pditommaso Aug 2, 2023
05c8c58
Merge branch 'master' into 3340-preview-container-images
pditommaso Aug 2, 2023
0db293f
Rename to PreviewContainersObserver [ci fast]
pditommaso Aug 2, 2023
938c7f3
Fix minor type error [ci skip]
pditommaso Aug 2, 2023
6f72b30
Revert unrelated changes
bentsherman Aug 2, 2023
d8a704f
Format JSON as array of objects
bentsherman Aug 2, 2023
631f38b
Pushing missing log [ci skip]
pditommaso Aug 2, 2023
4f1642a
Replace file output with stdout, suppress console output
bentsherman Aug 4, 2023
050874b
Update Wave and Fusion docs (#4149)
bentsherman Aug 3, 2023
4838bfb
Fix strict mode docs (#4150)
bentsherman Aug 3, 2023
6f65591
Fix bug with K8s resource labels (#4147) [ci fast]
bentsherman Aug 3, 2023
b009ecf
Add `-value` option to `config` command (#4142)
afishman Aug 3, 2023
f738ec0
Fix Wave disable flag
pditommaso Aug 4, 2023
7be17d3
Restore Tower CacheManager for backward compatibility
pditommaso Aug 5, 2023
36d6de3
Increase Wave client max attempts
pditommaso Aug 5, 2023
dd7d922
Bump [email protected]
pditommaso Aug 5, 2023
3555e00
Bump [email protected]
pditommaso Aug 5, 2023
167de54
Bump [email protected]
pditommaso Aug 5, 2023
ac2c473
Bump [email protected]
pditommaso Aug 5, 2023
2954436
Update changelog
pditommaso Aug 5, 2023
a391a75
[release 23.08.0-edge] Update timestamp and build number [ci fast]
pditommaso Aug 5, 2023
822184f
Enable cloud cache based on environment variable (#4160)
bentsherman Aug 8, 2023
2079efc
Minor changes [ci fast]
pditommaso Aug 9, 2023
8f0fa99
Fix typos in source code comments (#4173) [ci fast]
mribeirodantas Aug 10, 2023
8482c35
Add 429 http status code to Wave retriable errors
pditommaso Aug 10, 2023
d5ec1f0
Update changelog
pditommaso Aug 11, 2023
bd026e5
Improve Conda build error report [ci fast]
pditommaso Aug 13, 2023
da6ceba
Fix Process hangs when using flatten and finish errorStrategy
pditommaso Aug 13, 2023
1eb5293
Fix Execution should fail if report or timeline file already exists
pditommaso Aug 13, 2023
901ceb7
Fix env output when changing task workdir
pditommaso Aug 13, 2023
41385fa
Improve handling of name-only container env variables
pditommaso Aug 13, 2023
3a3fbf0
Document error about trailing backslash with space (#4180)
bentsherman Aug 13, 2023
ce01de2
Update AWS instructions for creating a custom AMI [ci skip] (#4174)
manuelesimi Aug 13, 2023
df20ac4
Remove lock file from cloudcache (#4167)
bentsherman Aug 13, 2023
f04f62c
Fix checkpoint thread termination (#4166)
pditommaso Aug 13, 2023
333ea7e
Update tip about modifying maps (#4153) [ci skip]
bentsherman Aug 14, 2023
2b13235
Apply K8s Pod metadata to Job (#4057) [ci fast]
bentsherman Aug 14, 2023
694cff0
Remove dockerize launcher classpath file (#4191) [ci fast]
bentsherman Aug 15, 2023
cec558c
Fix `workflow.container` map resolution (#4190)
bentsherman Aug 15, 2023
062cb1c
Escape semicolons in paths (#4193)
bentsherman Aug 15, 2023
7a26e4a
Use root user in Wave container based on micromamba (#4038) [ci fast]
marcodelapierre Aug 15, 2023
109f3b1
FIx nested InvocationTargetException (#4192) [ci fast]
bentsherman Aug 15, 2023
8b2c276
Fix if-guard on log.trace in trask processor inner class [ci fast]
pditommaso Aug 16, 2023
42ea6c8
Add resource labels support for Azure Batch (#4178)
bentsherman Aug 17, 2023
eb38aea
Bump [email protected]
pditommaso Aug 17, 2023
8223dbe
Bump [email protected]
pditommaso Aug 17, 2023
a70ec26
Bump [email protected]
pditommaso Aug 17, 2023
24a695d
Bump [email protected]
pditommaso Aug 17, 2023
ac72bc9
Update changelog [ci fast]
pditommaso Aug 17, 2023
d7c4fc1
[release 23.08.1-edge] Update timestamp and build number [ci fast]
pditommaso Aug 17, 2023
d2ac7d6
Fix security deps in nf-azure plugin
pditommaso Aug 17, 2023
513e0d2
Add support for Wave native build for singularity
pditommaso Aug 20, 2023
8ee9cb2
Bump groovy 3.0.19
pditommaso Aug 23, 2023
e69ec67
Move containers preview to its own command
pditommaso Aug 23, 2023
006623d
Merge branch 'master' into 3340-preview-container-images
pditommaso Aug 23, 2023
d549a82
apply suggestions from review, minor edits
bentsherman Aug 23, 2023
544235b
Add tests [ci fast]
pditommaso Aug 23, 2023
0481202
Fix & more
pditommaso Aug 23, 2023
44614b4
Just a blank [ci skip]
pditommaso Aug 23, 2023
a6c12b2
remove extraneous tests
bentsherman Aug 23, 2023
314f348
Add tests [ci fast]
pditommaso Aug 24, 2023
7ec45eb
Add prompt confirmation [ci fast]
pditommaso Aug 24, 2023
8f0eb45
Restructure json result
pditommaso Aug 24, 2023
15659cf
Add Wave dry-run mode + inspect -concretize
pditommaso Aug 24, 2023
dfdeeb4
update docs [ci fast]
bentsherman Aug 23, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions modules/nextflow/src/main/groovy/nextflow/Session.groovy
Original file line number Diff line number Diff line change
Expand Up @@ -468,13 +468,27 @@ class Session implements ISession {
CH.broadcast()

if( preview ) {
dumpContainerImages()
terminated = true
}
else {
callIgniters()
}
}

private void dumpContainerImages() {
for( def vertex : dag.vertices ) {
// skip nodes that are not processes
if( !vertex.process )
continue

// print process name and container image
def process = vertex.process
log.debug "Querying container image for process `${process.name}`..."
log.info "${process.name} ${process.getContainer()}"
}
}

private void callIgniters() {
log.debug "Igniting dataflow network (${igniters.size()})"
for( Closure action : igniters ) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -367,6 +367,23 @@ class TaskProcessor {

boolean hasErrors() { errorCount>0 }

/**
* Get the container image of the process, or return
* null if it can't be resolved.
*/
String getContainer() {
try {
def config = config.createTaskConfig()
config.context = new TaskContext(this)

return config.getContainer()
}
catch( Exception e ) {
log.warn1 "Unable to resolve container image for process `${name}`", e
return null
}
}

protected void checkWarn(String msg, Map opts=null) {
if( NF.isStrictMode() )
throw new ProcessUnrecoverableException(msg)
Expand Down