"CannotStartContainerError: Error response from daemon: failed to create task for container" when using seqera containers #5659

Shians · 2025-01-09T01:03:30Z

Shians
Jan 9, 2025

Bug report

I am very new to using Nextflow and AWS, following the basic settings I am able to run a "Hello world!" process successfully on EC2 backed AWS Batch. However when I try to use a seqera container I am unable to launch any jobs.

Steps to reproduce the problem

process HelloWorld {
    container 'community.wave.seqera.io/library/bcftools_pip_awscli:cae216d906baa276'
    // container 'amazonlinux:2023'

    tag "hello"
    label 'small' // cpus: 1, memory: 8.GB, time: 48h

    output:
    stdout

    script:
    """
    echo 'Hello, world!'
    """
}

Program output

ERROR ~ Error executing process > 'HelloWorld (hello)'

Caused by:
  Task failed to start - CannotStartContainerError: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/usr/local/bin/_entrypoint.sh": stat /usr/loca


Command executed:

  echo 'Hello, world!'

Command exit status:
  -

Command output:
  (empty)

Work dir:
<REDACTED>

Container:
  wave.seqera.io/wt/b3e1ad0ff50e/library/bcftools_pip_awscli:cae216d906baa276

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details

Environment

Nextflow version: 24.10.3 build 5933
Java version: openjdk 21 2023-09-19
Operating system: macOS
Bash version: zsh 5.9 (x86_64-apple-darwin23.0)

pditommaso · 2025-01-09T09:06:15Z

pditommaso
Jan 9, 2025
Maintainer

It looks like you have Wave enabled in your config. Since the container already exists, that's not needed.

9 replies

Shians Jan 13, 2025
Author

Jan-13 17:38:39.168 [main] DEBUG nextflow.cli.Launcher - $> nextflow run main.nf -profile aws
Jan-13 17:38:39.320 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W  ~  version 24.10.3
Jan-13 17:38:40.223 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/Users/su.s/.nextflow/plugins; core-plugins: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
Jan-13 17:38:40.264 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jan-13 17:38:40.265 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jan-13 17:38:40.267 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.12.0 in 'deployment' mode
Jan-13 17:38:40.288 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Jan-13 17:38:40.305 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /Users/su.s/Programs/NextFlow/AWS/nextflow.config
Jan-13 17:38:40.308 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /Users/su.s/Programs/NextFlow/AWS/nextflow.config
Jan-13 17:38:40.334 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /Users/su.s/.nextflow/secrets/store.json
Jan-13 17:38:40.336 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@29d37757] - activable => nextflow.secret.LocalSecretsProvider@29d37757
Jan-13 17:38:40.339 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `aws`
Jan-13 17:38:40.737 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [aws]
Jan-13 17:38:40.760 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
Jan-13 17:38:40.771 [main] DEBUG nextflow.cli.CmdRun - Launching `main.nf` [lethal_minsky] DSL2 - revision: 15e929f35f
Jan-13 17:38:40.772 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[[email protected]]
Jan-13 17:38:40.772 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[[email protected]]
Jan-13 17:38:40.772 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-amazon version: 2.9.2
Jan-13 17:38:40.796 [main] INFO  org.pf4j.AbstractPluginManager - Plugin '[email protected]' resolved
Jan-13 17:38:40.796 [main] INFO  org.pf4j.AbstractPluginManager - Start plugin '[email protected]'
Jan-13 17:38:40.888 [main] DEBUG nextflow.plugin.BasePlugin - Plugin started [email protected]
Jan-13 17:38:40.900 [main] DEBUG nextflow.file.FileHelper - > Added 'S3FileSystemProvider' to list of installed providers [s3]
Jan-13 17:38:40.942 [main] DEBUG nextflow.Session - Session UUID: 48bd3894-dccc-47fb-876b-a1055c538e3e
Jan-13 17:38:40.942 [main] DEBUG nextflow.Session - Run name: lethal_minsky
Jan-13 17:38:40.942 [main] DEBUG nextflow.Session - Executor pool size: 11
Jan-13 17:38:40.953 [main] DEBUG nextflow.file.FileHelper - Creating a file system instance for provider: S3FileSystemProvider
Jan-13 17:38:40.965 [main] DEBUG nextflow.cloud.aws.config.AwsConfig - AWS S3 config properties: {max_error_retry=5}
Jan-13 17:38:41.148 [main] DEBUG nextflow.cloud.aws.AwsClientFactory - Merging AWS credentials file '/Users/su.s/.aws/credentials' and config file '/Users/su.s/.aws/config'
Jan-13 17:38:41.482 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Jan-13 17:38:41.489 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=33; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jan-13 17:38:41.540 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 24.10.3 build 5933
  Created: 16-12-2024 15:34 UTC (17-12-2024 02:34 AEDT)
  System: Mac OS X 14.6.1
  Runtime: Groovy 4.0.23 on OpenJDK 64-Bit Server VM 21+35
  Encoding: UTF-8 (UTF-8)
  Process: 33449@mac4183 [192.168.50.101]
  CPUs: 11 - Mem: 36 GB (109.8 MB) - Swap: 16 GB (829.4 MB)
Jan-13 17:38:43.394 [main] DEBUG nextflow.Session - Work-dir: <REDACTED> [Mac OS X]
Jan-13 17:38:43.394 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /Users/su.s/Programs/NextFlow/AWS/bin
Jan-13 17:38:43.414 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[AwsBatchExecutor]
Jan-13 17:38:43.423 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jan-13 17:38:43.440 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jan-13 17:38:43.453 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 12; maxThreads: 1000
Jan-13 17:38:43.491 [main] DEBUG nextflow.Session - Session start
Jan-13 17:38:43.760 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jan-13 17:38:43.817 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:small` matches labels `small` for process with name HelloWorld
Jan-13 17:38:43.822 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: awsbatch
Jan-13 17:38:43.823 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'awsbatch'
Jan-13 17:38:43.825 [main] DEBUG nextflow.executor.Executor - [warm up] executor > awsbatch
Jan-13 17:38:43.836 [main] DEBUG nextflow.util.ThrottlingExecutor - Creating throttling executor with opts: nextflow.util.ThrottlingExecutor$Options(poolName:AWSBatch-executor, limiter:RateLimiter[stableRate=50.0qps], poolSize:55, maxPoolSize:55, queueSize:5000, maxRetries:10, keepAlive:1m, autoThrottle:true, errorBurstDelay:1s, rampUpInterval:100, rampUpFactor:1.2, rampUpMaxRate:1.7976931348623157E308, backOffFactor:2.0, backOffMinRate:0.0166666667, retryDelay:1s)
Jan-13 17:38:43.840 [main] DEBUG nextflow.util.ThrottlingExecutor - Creating throttling executor with opts: nextflow.util.ThrottlingExecutor$Options(poolName:AWSBatch-reaper, limiter:RateLimiter[stableRate=50.0qps], poolSize:55, maxPoolSize:55, queueSize:5000, maxRetries:10, keepAlive:1m, autoThrottle:true, errorBurstDelay:1s, rampUpInterval:100, rampUpFactor:1.2, rampUpMaxRate:1.7976931348623157E308, backOffFactor:2.0, backOffMinRate:0.0166666667, retryDelay:1s)
Jan-13 17:38:43.841 [main] DEBUG n.cloud.aws.batch.AwsBatchExecutor - Creating parallel monitor for executor 'awsbatch' > pollInterval=10s; dumpInterval=5m
Jan-13 17:38:43.848 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: awsbatch)
Jan-13 17:38:43.850 [main] DEBUG nextflow.cloud.aws.AwsClientFactory - Merging AWS credentials file '/Users/su.s/.aws/credentials' and config file '/Users/su.s/.aws/config'
Jan-13 17:38:43.908 [main] DEBUG n.cloud.aws.batch.AwsBatchExecutor - [AWS BATCH] Executor options=AwsOptions(remoteBinDir:null, fetchInstanceType:false, region:us-west-2, jobRole:arn:aws:iam::260571314817:role/NextflowBatchAgent, logsGroup:null, retryMode:standard, shareIdentifier:null, schedulingPriority:0, cliPath:/usr/bin/aws, volumes:[], maxParallelTransfers:4, maxTransferAttempts:5, maxSpotAttempts:0, delayBetweenAttempts:10s, storageClass:null, storageEncryption:null, storageKmsKeyId:null, s3Acl:null, debug:null, requesterPays:null, awsCli:/usr/bin/aws --region us-west-2, fargateMode:false, s5cmdPath:null, executionRole:arn:aws:iam::260571314817:role/NextflowBatchAgent)
Jan-13 17:38:43.972 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: FilterVariants, HelloWorld, DownloadFiles
Jan-13 17:38:43.973 [main] DEBUG nextflow.Session - Igniting dataflow network (2)
Jan-13 17:38:43.973 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > HelloWorld
Jan-13 17:38:43.975 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
  Script_bff5ed2717da797a: /Users/su.s/Programs/NextFlow/AWS/main.nf
Jan-13 17:38:43.975 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination 
Jan-13 17:38:43.975 [main] DEBUG nextflow.Session - Session await
Jan-13 17:38:44.021 [Actor Thread 4] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=HelloWorld; work-dir=null
  error [java.io.IOException]: Cannot store Conda environments to a remote work directory -- Use a POSIX compatible work directory or specify an alternative path with the `conda.cacheDir` config setting
Jan-13 17:38:44.028 [Actor Thread 4] ERROR nextflow.processor.TaskProcessor - Error executing process > 'HelloWorld'

Caused by:
  Cannot store Conda environments to a remote work directory -- Use a POSIX compatible work directory or specify an alternative path with the `conda.cacheDir` config setting


java.io.IOException: Cannot store Conda environments to a remote work directory -- Use a POSIX compatible work directory or specify an alternative path with the `conda.cacheDir` config setting
	at nextflow.conda.CondaCache.getCacheDir(CondaCache.groovy:146)
	at nextflow.conda.CondaCache.condaPrefixPath(CondaCache.groovy:244)
	at nextflow.conda.CondaCache.getLazyImagePath(CondaCache.groovy:363)
	at nextflow.conda.CondaCache.getCachePathFor(CondaCache.groovy:393)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.processor.TaskRun.getCondaEnv0(TaskRun.groovy:647)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnCurrentN(ScriptBytecodeAdapter.java:96)
	at nextflow.processor.TaskRun.this$dist$invoke$1(TaskRun.groovy)
	at nextflow.processor.TaskRun$2.methodMissing(TaskRun.groovy)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at groovy.lang.MetaClassImpl.invokeMissingMethod(MetaClassImpl.java:924)
	at groovy.lang.MetaClassImpl.invokePropertyOrMissing(MetaClassImpl.java:1413)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1335)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.processor.TaskRun$2.apply(TaskRun.groovy:638)
	at nextflow.processor.TaskRun$2.apply(TaskRun.groovy)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.processor.TaskRun.getCondaEnv(TaskRun.groovy:635)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.processor.TaskProcessor.createTaskHashKey(TaskProcessor.groovy:2224)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.processor.TaskProcessor.invokeTask(TaskProcessor.groovy:654)
	at nextflow.processor.InvokeTaskAdapter.call(InvokeTaskAdapter.groovy:52)
	at groovyx.gpars.dataflow.operator.DataflowOperatorActor.startTask(DataflowOperatorActor.java:120)
	at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor.access$001(ForkingDataflowOperatorActor.java:35)
	at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor$1.run(ForkingDataflowOperatorActor.java:58)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Jan-13 17:38:44.032 [Actor Thread 4] DEBUG nextflow.Session - Session aborted -- Cause: Cannot store Conda environments to a remote work directory -- Use a POSIX compatible work directory or specify an alternative path with the `conda.cacheDir` config setting
Jan-13 17:38:44.055 [Actor Thread 4] DEBUG nextflow.Session - The following nodes are still active:
[process] HelloWorld
  status=ACTIVE
  port 0: (cntrl) -     ; channel: $

Jan-13 17:38:44.060 [Actor Thread 4] DEBUG nextflow.Session - 
Thread[#37,Actor Thread 1,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack$SNode.block(SynchronousQueue.java:288)
  java.base@21/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
  java.base@21/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:397)
  java.base@21/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:886)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#36,Task submitter,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519)
  java.base@21/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
  java.base@21/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1707)
  app//nextflow.processor.TaskPollingMonitor.awaitTasks(TaskPollingMonitor.groovy:388)
  app//nextflow.processor.TaskPollingMonitor.submitLoop(TaskPollingMonitor.groovy:412)
  java.base@21/java.lang.invoke.LambdaForm$DMH/0x00000008011ac000.invokeVirtual(LambdaForm$DMH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801528000.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:153)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  java.base@21/java.lang.reflect.Method.invoke(Method.java:580)
  app//org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
  app//groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethodClosure(MetaClassImpl.java:1017)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1207)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
  app//groovy.lang.Closure.call(Closure.java:433)
  app//groovy.lang.Closure.call(Closure.java:412)
  app//groovy.lang.Closure.run(Closure.java:505)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#34,AnsiLogObserver,5,main]
  java.base@21/java.lang.Object.wait0(Native Method)
  java.base@21/java.lang.Object.wait(Object.java:366)
  app//nextflow.trace.AnsiLogObserver.render0(AnsiLogObserver.groovy:185)
  java.base@21/java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(DirectMethodHandle$Holder)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x000000080159c800.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:154)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  java.base@21/java.lang.reflect.Method.invoke(Method.java:580)
  app//org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
  app//groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethodClosure(MetaClassImpl.java:1017)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1207)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
  app//groovy.lang.Closure.call(Closure.java:433)
  app//groovy.lang.Closure.call(Closure.java:412)
  app//groovy.lang.Closure.run(Closure.java:505)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#40,Actor Thread 4,5,main]
  java.base@21/java.lang.Thread.dumpThreads(Native Method)
  java.base@21/java.lang.Thread.getAllStackTraces(Thread.java:2521)
  app//nextflow.util.SysHelper.dumpThreads(SysHelper.groovy:188)
  app//nextflow.Session.abort(Session.groovy:800)
  app//nextflow.Session.fault(Session.groovy:766)
  app//nextflow.Session.fault(Session.groovy)
  java.base@21/java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(DirectMethodHandle$Holder)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x000000080120d000.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801102c00.guardWithCatch(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801103400.reinvoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801103800.guard(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801103400.reinvoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801103800.guard(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801103400.reinvoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801163c00.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
  app//org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
  java.base@21/java.lang.invoke.LambdaForm$DMH/0x00000008010aa800.invokeStatic(LambdaForm$DMH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801102400.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
  app//nextflow.processor.TaskProcessor.handleException(TaskProcessor.groovy:1004)
  java.base@21/java.lang.invoke.LambdaForm$DMH/0x00000008010ba800.invokeSpecial(LambdaForm$DMH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801638400.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801182800.invokeExact_MT(LambdaForm$MH)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:155)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  java.base@21/java.lang.reflect.Method.invoke(Method.java:580)
  app//org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
  app//groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnCurrentN(ScriptBytecodeAdapter.java:96)
  app//nextflow.processor.TaskProcessor.this$dist$invoke$1(TaskProcessor.groovy)
  app//nextflow.processor.TaskProcessor$TaskProcessorInterceptor.methodMissing(TaskProcessor.groovy)
  java.base@21/java.lang.invoke.LambdaForm$DMH/0x00000008010b2c00.invokeVirtual(LambdaForm$DMH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x000000080120ac00.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801182800.invokeExact_MT(LambdaForm$MH)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:155)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  java.base@21/java.lang.reflect.Method.invoke(Method.java:580)
  app//org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
  app//groovy.lang.MetaClassImpl.invokeMissingMethod(MetaClassImpl.java:924)
  app//groovy.lang.MetaClassImpl.invokePropertyOrMissing(MetaClassImpl.java:1413)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1335)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
  java.base@21/java.lang.invoke.LambdaForm$DMH/0x00000008010b3800.invokeInterface(LambdaForm$DMH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801179400.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000008012e8400.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801161400.guardWithCatch(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801162000.reinvoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801162400.guard(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801162000.reinvoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801162400.guard(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801162000.reinvoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000008011a1800.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
  app//org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
  java.base@21/java.lang.invoke.LambdaForm$DMH/0x00000008010aa800.invokeStatic(LambdaForm$DMH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801160800.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
  app//nextflow.processor.TaskProcessor$TaskProcessorInterceptor.onException(TaskProcessor.groovy:2606)
  app//groovyx.gpars.dataflow.operator.DataflowProcessor.fireOnException(DataflowProcessor.java:331)
  app//groovyx.gpars.dataflow.operator.DataflowProcessor.reportError(DataflowProcessor.java:293)
  app//groovyx.gpars.dataflow.operator.DataflowProcessorActor.reportException(DataflowProcessorActor.java:125)
  app//groovyx.gpars.dataflow.operator.DataflowOperatorActor.startTask(DataflowOperatorActor.java:122)
  app//groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor.access$001(ForkingDataflowOperatorActor.java:35)
  app//groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor$1.run(ForkingDataflowOperatorActor.java:58)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#33,Thread-1,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1758)
  java.base@21/java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:515)
  java.base@21/java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:677)
  app//nextflow.util.SimpleAgent.run(SimpleAgent.groovy:89)
  java.base@21/java.lang.invoke.LambdaForm$DMH/0x00000008011ac000.invokeVirtual(LambdaForm$DMH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801528000.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:153)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  java.base@21/java.lang.reflect.Method.invoke(Method.java:580)
  app//org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
  app//groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethodClosure(MetaClassImpl.java:1017)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1207)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
  app//groovy.lang.Closure.call(Closure.java:433)
  app//groovy.lang.Closure.call(Closure.java:412)
  app//groovy.lang.Closure.run(Closure.java:505)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#32,java-sdk-http-connection-reaper,5,main]
  java.base@21/java.lang.Thread.sleep0(Native Method)
  java.base@21/java.lang.Thread.sleep(Thread.java:509)
  com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:188)

Thread[#39,Actor Thread 3,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack$SNode.block(SynchronousQueue.java:288)
  java.base@21/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
  java.base@21/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:397)
  java.base@21/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:886)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#18,Common-Cleaner,8,InnocuousThreadGroup]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1847)
  java.base@21/java.lang.ref.ReferenceQueue.await(ReferenceQueue.java:71)
  java.base@21/java.lang.ref.ReferenceQueue.remove0(ReferenceQueue.java:143)
  java.base@21/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:218)
  java.base@21/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:140)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)
  java.base@21/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:186)

Thread[#9,Reference Handler,10,system]
  java.base@21/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  java.base@21/java.lang.ref.Reference.processPendingReferences(Reference.java:246)
  java.base@21/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:208)

Thread[#19,Notification Thread,9,system]

Thread[#10,Finalizer,8,system]
  java.base@21/java.lang.Object.wait0(Native Method)
  java.base@21/java.lang.Object.wait(Object.java:366)
  java.base@21/java.lang.Object.wait(Object.java:339)
  java.base@21/java.lang.ref.NativeReferenceQueue.await(NativeReferenceQueue.java:48)
  java.base@21/java.lang.ref.ReferenceQueue.remove0(ReferenceQueue.java:158)
  java.base@21/java.lang.ref.NativeReferenceQueue.remove(NativeReferenceQueue.java:89)
  java.base@21/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:173)

Thread[#38,Actor Thread 2,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:754)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1099)
  java.base@21/java.util.concurrent.Semaphore.acquire(Semaphore.java:475)
  app//groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor.forwardPoisonPill(ForkingDataflowOperatorActor.java:69)
  app//groovyx.gpars.dataflow.operator.DataflowProcessorActor.checkPoison(DataflowProcessorActor.java:114)
  app//groovyx.gpars.dataflow.operator.DataflowOperatorActor.onMessage(DataflowOperatorActor.java:83)
  app//groovyx.gpars.actor.impl.SDAClosure$1.call(SDAClosure.java:43)
  app//groovyx.gpars.actor.AbstractLoopingActor.runEnhancedWithoutRepliesOnMessages(AbstractLoopingActor.java:293)
  app//groovyx.gpars.actor.AbstractLoopingActor.access$400(AbstractLoopingActor.java:30)
  app//groovyx.gpars.actor.AbstractLoopingActor$1.handleMessage(AbstractLoopingActor.java:93)
  app//groovyx.gpars.util.AsyncMessagingCore.run(AsyncMessagingCore.java:132)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#29,process reaper,10,InnocuousThreadGroup]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
  java.base@21/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)
  java.base@21/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:186)

Thread[#1,main,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519)
  java.base@21/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
  java.base@21/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1707)
  app//nextflow.util.Barrier$_awaitCompletion_closure3.doCall(Barrier.groovy:92)
  app//nextflow.util.Barrier$_awaitCompletion_closure3.call(Barrier.groovy)
  app//nextflow.extension.Bolts.withLock(Bolts.groovy:337)
  app//nextflow.extension.Bolts.withLock(Bolts.groovy)
  app//nextflow.util.Barrier.awaitCompletion(Barrier.groovy:89)
  app//nextflow.Session.await(Session.groovy:676)
  app//nextflow.script.ScriptRunner.await(ScriptRunner.groovy:256)
  app//nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:144)
  app//nextflow.cli.CmdRun.run(CmdRun.groovy:376)
  app//nextflow.cli.Launcher.run(Launcher.groovy:503)
  app//nextflow.cli.Launcher.main(Launcher.groovy:658)

Thread[#11,Signal Dispatcher,9,system]

Thread[#35,Task monitor,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1847)
  app//nextflow.processor.TaskPollingMonitor$_await_closure5.doCall(TaskPollingMonitor.groovy:529)
  app//nextflow.processor.TaskPollingMonitor$_await_closure5.call(TaskPollingMonitor.groovy)
  app//nextflow.extension.Bolts.withLock(Bolts.groovy:337)
  app//nextflow.extension.Bolts.withLock(Bolts.groovy)
  app//nextflow.processor.TaskPollingMonitor.await(TaskPollingMonitor.groovy:528)
  app//nextflow.processor.TaskPollingMonitor.pollLoop(TaskPollingMonitor.groovy:453)
  java.base@21/java.lang.invoke.LambdaForm$DMH/0x00000008011ac000.invokeVirtual(LambdaForm$DMH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000000801528000.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:153)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  java.base@21/java.lang.reflect.Method.invoke(Method.java:580)
  app//org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
  app//groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
  app//org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:645)
  app//org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:628)
  app//org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:82)
  app//nextflow.processor.TaskPollingMonitor$_start_closure2.doCall(TaskPollingMonitor.groovy:318)
  app//nextflow.processor.TaskPollingMonitor$_start_closure2.call(TaskPollingMonitor.groovy)
  app//groovy.lang.Closure.run(Closure.java:505)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#41,Actor Thread 5,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack$SNode.block(SynchronousQueue.java:288)
  java.base@21/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
  java.base@21/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:397)
  java.base@21/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:886)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Jan-13 17:38:44.064 [main] DEBUG nextflow.Session - Session await > all processes finished
Jan-13 17:38:44.064 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jan-13 17:38:44.065 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: awsbatch) - terminating tasks monitor poll loop
Jan-13 17:38:44.067 [Actor Thread 2] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=HelloWorld; work-dir=null
  error [java.lang.InterruptedException]: java.lang.InterruptedException
Jan-13 17:38:44.071 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=0ms; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=0; peakCpus=0; peakMemory=0; ]
Jan-13 17:38:44.293 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Jan-13 17:38:44.309 [main] INFO  org.pf4j.AbstractPluginManager - Stop plugin '[email protected]'
Jan-13 17:38:44.309 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-amazon
Jan-13 17:38:44.310 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

pditommaso Jan 13, 2025
Maintainer

This log tells you are trying to use Conda with AWS Batch that's not supported (because the environment would build locally and the job execute remotely), and it's not related to the error reported as subject of this thread

Shians Jan 13, 2025
Author

Apologies, that was the wrong log when I tried a different approach

Jan-13 19:25:14.280 [main] DEBUG nextflow.cli.Launcher - $> nextflow run main.nf -profile aws
Jan-13 19:25:14.415 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W  ~  version 24.10.3
Jan-13 19:25:14.432 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/Users/su.s/.nextflow/plugins; core-plugins: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
Jan-13 19:25:14.469 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jan-13 19:25:14.470 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jan-13 19:25:14.472 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.12.0 in 'deployment' mode
Jan-13 19:25:14.486 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Jan-13 19:25:14.504 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /Users/su.s/Programs/NextFlow/AWS/nextflow.config
Jan-13 19:25:14.506 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /Users/su.s/Programs/NextFlow/AWS/nextflow.config
Jan-13 19:25:14.530 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /Users/su.s/.nextflow/secrets/store.json
Jan-13 19:25:14.532 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@39ce27f2] - activable => nextflow.secret.LocalSecretsProvider@39ce27f2
Jan-13 19:25:14.536 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `aws`
Jan-13 19:25:14.884 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [aws]
Jan-13 19:25:14.905 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
Jan-13 19:25:14.919 [main] DEBUG nextflow.cli.CmdRun - Launching `main.nf` [thirsty_mayer] DSL2 - revision: 00275bb373
Jan-13 19:25:14.920 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[[email protected]]
Jan-13 19:25:14.920 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[[email protected]]
Jan-13 19:25:14.920 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-amazon version: 2.9.2
Jan-13 19:25:14.953 [main] INFO  org.pf4j.AbstractPluginManager - Plugin '[email protected]' resolved
Jan-13 19:25:14.953 [main] INFO  org.pf4j.AbstractPluginManager - Start plugin '[email protected]'
Jan-13 19:25:15.018 [main] DEBUG nextflow.plugin.BasePlugin - Plugin started [email protected]
Jan-13 19:25:15.037 [main] DEBUG nextflow.file.FileHelper - > Added 'S3FileSystemProvider' to list of installed providers [s3]
Jan-13 19:25:15.072 [main] DEBUG nextflow.Session - Session UUID: 1733b593-406e-4db9-8ece-ee2cd57e80c6
Jan-13 19:25:15.073 [main] DEBUG nextflow.Session - Run name: thirsty_mayer
Jan-13 19:25:15.073 [main] DEBUG nextflow.Session - Executor pool size: 11
Jan-13 19:25:15.084 [main] DEBUG nextflow.file.FileHelper - Creating a file system instance for provider: S3FileSystemProvider
Jan-13 19:25:15.093 [main] DEBUG nextflow.cloud.aws.config.AwsConfig - AWS S3 config properties: {max_error_retry=5}
Jan-13 19:25:15.254 [main] DEBUG nextflow.cloud.aws.AwsClientFactory - Merging AWS credentials file '/Users/su.s/.aws/credentials' and config file '/Users/su.s/.aws/config'
Jan-13 19:25:15.598 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Jan-13 19:25:15.603 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=33; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jan-13 19:25:15.674 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 24.10.3 build 5933
  Created: 16-12-2024 15:34 UTC (17-12-2024 02:34 AEDT)
  System: Mac OS X 14.6.1
  Runtime: Groovy 4.0.23 on OpenJDK 64-Bit Server VM 21+35
  Encoding: UTF-8 (UTF-8)
  Process: 43371@mac4183 [192.168.50.101]
  CPUs: 11 - Mem: 36 GB (82.8 MB) - Swap: 19 GB (729.6 MB)
Jan-13 19:25:17.179 [main] DEBUG nextflow.Session - Work-dir: <REDACTED> [Mac OS X]
Jan-13 19:25:17.179 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /Users/su.s/Programs/NextFlow/AWS/bin
Jan-13 19:25:17.197 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[AwsBatchExecutor]
Jan-13 19:25:17.204 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jan-13 19:25:17.221 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jan-13 19:25:17.234 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 12; maxThreads: 1000
Jan-13 19:25:17.273 [main] DEBUG nextflow.Session - Session start
Jan-13 19:25:17.484 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jan-13 19:25:17.541 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:small` matches labels `small` for process with name HelloWorld
Jan-13 19:25:17.545 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: awsbatch
Jan-13 19:25:17.545 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'awsbatch'
Jan-13 19:25:17.547 [main] DEBUG nextflow.executor.Executor - [warm up] executor > awsbatch
Jan-13 19:25:17.554 [main] DEBUG nextflow.util.ThrottlingExecutor - Creating throttling executor with opts: nextflow.util.ThrottlingExecutor$Options(poolName:AWSBatch-executor, limiter:RateLimiter[stableRate=50.0qps], poolSize:55, maxPoolSize:55, queueSize:5000, maxRetries:10, keepAlive:1m, autoThrottle:true, errorBurstDelay:1s, rampUpInterval:100, rampUpFactor:1.2, rampUpMaxRate:1.7976931348623157E308, backOffFactor:2.0, backOffMinRate:0.0166666667, retryDelay:1s)
Jan-13 19:25:17.558 [main] DEBUG nextflow.util.ThrottlingExecutor - Creating throttling executor with opts: nextflow.util.ThrottlingExecutor$Options(poolName:AWSBatch-reaper, limiter:RateLimiter[stableRate=50.0qps], poolSize:55, maxPoolSize:55, queueSize:5000, maxRetries:10, keepAlive:1m, autoThrottle:true, errorBurstDelay:1s, rampUpInterval:100, rampUpFactor:1.2, rampUpMaxRate:1.7976931348623157E308, backOffFactor:2.0, backOffMinRate:0.0166666667, retryDelay:1s)
Jan-13 19:25:17.559 [main] DEBUG n.cloud.aws.batch.AwsBatchExecutor - Creating parallel monitor for executor 'awsbatch' > pollInterval=10s; dumpInterval=5m
Jan-13 19:25:17.563 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: awsbatch)
Jan-13 19:25:17.565 [main] DEBUG nextflow.cloud.aws.AwsClientFactory - Merging AWS credentials file '/Users/su.s/.aws/credentials' and config file '/Users/su.s/.aws/config'
Jan-13 19:25:17.590 [main] DEBUG n.cloud.aws.batch.AwsBatchExecutor - [AWS BATCH] Executor options=AwsOptions(remoteBinDir:null, fetchInstanceType:false, region:us-west-2, jobRole:arn:aws:iam::260571314817:role/NextflowBatchAgent, logsGroup:null, retryMode:standard, shareIdentifier:null, schedulingPriority:0, cliPath:/usr/bin/aws, volumes:[], maxParallelTransfers:4, maxTransferAttempts:5, maxSpotAttempts:0, delayBetweenAttempts:10s, storageClass:null, storageEncryption:null, storageKmsKeyId:null, s3Acl:null, debug:null, requesterPays:null, awsCli:/usr/bin/aws --region us-west-2, fargateMode:false, s5cmdPath:null, executionRole:arn:aws:iam::260571314817:role/NextflowBatchAgent)
Jan-13 19:25:17.636 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: FilterVariants, HelloWorld, DownloadFiles
Jan-13 19:25:17.636 [main] DEBUG nextflow.Session - Igniting dataflow network (2)
Jan-13 19:25:17.637 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > HelloWorld
Jan-13 19:25:17.639 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
  Script_519eae793aabbb30: /Users/su.s/Programs/NextFlow/AWS/main.nf
Jan-13 19:25:17.639 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination 
Jan-13 19:25:17.639 [main] DEBUG nextflow.Session - Session await
Jan-13 19:25:19.722 [AWSBatch-executor-1] DEBUG n.c.aws.batch.AwsBatchTaskHandler - [AWS BATCH] [AWS BATCH] Found job definition name=nf-community-wave-seqera-io-library-bcftools_pip_awscli-cae216d906baa276:1; container=community.wave.seqera.io/library/bcftools_pip_awscli:cae216d906baa276
Jan-13 19:25:20.008 [AWSBatch-executor-1] DEBUG n.c.aws.batch.AwsBatchTaskHandler - [AWS BATCH] Process `HelloWorld (hello)` submitted > job=09aa4c5d-2498-4642-bff3-9049b8264313; work-dir=<REDACTED>
Jan-13 19:25:20.009 [AWSBatch-executor-1] INFO  nextflow.Session - [8f/a2501d] Submitted process > HelloWorld (hello)
Jan-13 19:27:38.896 [Task monitor] DEBUG n.c.aws.batch.AwsBatchTaskHandler - [AWS BATCH] Cannot read exit status for task: `HelloWorld (hello)` | /nf-workdir-us-west-2/8f/a2501d01c31fb59109620f70554dea/.exitcode
Jan-13 19:27:39.160 [Task monitor] DEBUG nextflow.cloud.aws.AwsClientFactory - Merging AWS credentials file '/Users/su.s/.aws/credentials' and config file '/Users/su.s/.aws/config'
Jan-13 19:27:39.986 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: HelloWorld (hello); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Task failed to start - CannotStartContainerError: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/usr/local/bin/_entrypoint.sh": stat /usr/loca; workDir: <REDACTED>]
Jan-13 19:27:39.989 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'TaskFinalizer' minSize=10; maxSize=33; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jan-13 19:27:40.197 [TaskFinalizer-1] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=HelloWorld (hello); work-dir=<REDACTED>
  error [nextflow.exception.ProcessFailedException]: Process `HelloWorld (hello)` failed
Jan-13 19:27:40.411 [TaskFinalizer-1] DEBUG nextflow.processor.TaskRun - Unable to dump output of process 'HelloWorld (hello)' -- Cause: java.nio.file.NoSuchFileException: /var/folders/3p/qh68nr3j5h3054n7llf_9l1r00025j/T/temp-s3-16078291349125495528/.command.out
Jan-13 19:27:40.603 [TaskFinalizer-1] DEBUG nextflow.processor.TaskRun - Unable to dump error of process 'HelloWorld (hello)' -- Cause: java.nio.file.NoSuchFileException: /var/folders/3p/qh68nr3j5h3054n7llf_9l1r00025j/T/temp-s3-7316311295549192880/.command.err
Jan-13 19:27:40.614 [TaskFinalizer-1] ERROR nextflow.processor.TaskProcessor - Error executing process > 'HelloWorld (hello)'

Caused by:
  Task failed to start - CannotStartContainerError: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/usr/local/bin/_entrypoint.sh": stat /usr/loca


Command executed:

  echo 'Hello, world!'
  bcftools --version

Command exit status:
  -

Command output:
  (empty)

Work dir:
  <REDACTED>

Container:
  community.wave.seqera.io/library/bcftools_pip_awscli:cae216d906baa276

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
Jan-13 19:27:40.630 [main] DEBUG nextflow.Session - Session await > all processes finished
Jan-13 19:27:40.813 [TaskFinalizer-1] DEBUG nextflow.Session - Session aborted -- Cause: Process `HelloWorld (hello)` failed
Jan-13 19:27:40.882 [TaskFinalizer-1] DEBUG nextflow.Session - 
Thread[#34,Task monitor,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1847)
  app//nextflow.processor.TaskPollingMonitor$_await_closure5.doCall(TaskPollingMonitor.groovy:529)
  app//nextflow.processor.TaskPollingMonitor$_await_closure5.call(TaskPollingMonitor.groovy)
  app//nextflow.extension.Bolts.withLock(Bolts.groovy:337)
  app//nextflow.extension.Bolts.withLock(Bolts.groovy)
  app//nextflow.processor.TaskPollingMonitor.await(TaskPollingMonitor.groovy:528)
  app//nextflow.processor.TaskPollingMonitor.pollLoop(TaskPollingMonitor.groovy:453)
  java.base@21/java.lang.invoke.LambdaForm$DMH/0x00000070011ac000.invokeVirtual(LambdaForm$DMH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000007001520400.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:153)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  java.base@21/java.lang.reflect.Method.invoke(Method.java:580)
  app//org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
  app//groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
  app//org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:645)
  app//org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:628)
  app//org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:82)
  app//nextflow.processor.TaskPollingMonitor$_start_closure2.doCall(TaskPollingMonitor.groovy:318)
  app//nextflow.processor.TaskPollingMonitor$_start_closure2.call(TaskPollingMonitor.groovy)
  app//groovy.lang.Closure.run(Closure.java:505)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#59,process reaper,10,InnocuousThreadGroup]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)
  java.base@21/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)
  java.base@21/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:186)

Thread[#58,AWSBatch-executor-15,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1758)
  java.base@21/java.util.concurrent.PriorityBlockingQueue.poll(PriorityBlockingQueue.java:549)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#40,Actor Thread 5,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack$SNode.block(SynchronousQueue.java:288)
  java.base@21/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
  java.base@21/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:397)
  java.base@21/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:886)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#19,Notification Thread,9,system]

Thread[#10,Finalizer,8,system]
  java.base@21/java.lang.Object.wait0(Native Method)
  java.base@21/java.lang.Object.wait(Object.java:366)
  java.base@21/java.lang.Object.wait(Object.java:339)
  java.base@21/java.lang.ref.NativeReferenceQueue.await(NativeReferenceQueue.java:48)
  java.base@21/java.lang.ref.ReferenceQueue.remove0(ReferenceQueue.java:158)
  java.base@21/java.lang.ref.NativeReferenceQueue.remove(NativeReferenceQueue.java:89)
  java.base@21/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:173)

Thread[#44,Actor Thread 6,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack$SNode.block(SynchronousQueue.java:288)
  java.base@21/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
  java.base@21/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:397)
  java.base@21/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:886)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#11,Signal Dispatcher,9,system]

Thread[#57,AWSBatch-executor-14,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1758)
  java.base@21/java.util.concurrent.PriorityBlockingQueue.poll(PriorityBlockingQueue.java:549)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#31,java-sdk-http-connection-reaper,5,main]
  java.base@21/java.lang.Thread.sleep0(Native Method)
  java.base@21/java.lang.Thread.sleep(Thread.java:509)
  com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:188)

Thread[#33,AnsiLogObserver,5,main]
  java.base@21/java.lang.Object.wait0(Native Method)
  java.base@21/java.lang.Object.wait(Object.java:366)
  app//nextflow.trace.AnsiLogObserver.render0(AnsiLogObserver.groovy:185)
  java.base@21/java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(DirectMethodHandle$Holder)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x000000700159c800.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:154)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  java.base@21/java.lang.reflect.Method.invoke(Method.java:580)
  app//org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
  app//groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethodClosure(MetaClassImpl.java:1017)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1207)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
  app//groovy.lang.Closure.call(Closure.java:433)
  app//groovy.lang.Closure.call(Closure.java:412)
  app//groovy.lang.Closure.run(Closure.java:505)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#18,Common-Cleaner,8,InnocuousThreadGroup]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1847)
  java.base@21/java.lang.ref.ReferenceQueue.await(ReferenceQueue.java:71)
  java.base@21/java.lang.ref.ReferenceQueue.remove0(ReferenceQueue.java:143)
  java.base@21/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:218)
  java.base@21/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:140)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)
  java.base@21/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:186)

Thread[#39,Actor Thread 4,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack$SNode.block(SynchronousQueue.java:288)
  java.base@21/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
  java.base@21/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:397)
  java.base@21/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:886)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#55,AWSBatch-executor-12,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1758)
  java.base@21/java.util.concurrent.PriorityBlockingQueue.poll(PriorityBlockingQueue.java:549)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#32,Thread-1,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1758)
  java.base@21/java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:515)
  java.base@21/java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:677)
  app//nextflow.util.SimpleAgent.run(SimpleAgent.groovy:89)
  java.base@21/java.lang.invoke.LambdaForm$DMH/0x00000070011ac000.invokeVirtual(LambdaForm$DMH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000007001520400.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:153)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  java.base@21/java.lang.reflect.Method.invoke(Method.java:580)
  app//org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
  app//groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethodClosure(MetaClassImpl.java:1017)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1207)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
  app//groovy.lang.Closure.call(Closure.java:433)
  app//groovy.lang.Closure.call(Closure.java:412)
  app//groovy.lang.Closure.run(Closure.java:505)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#60,TaskFinalizer-1,5,main]
  java.base@21/java.lang.Thread.dumpThreads(Native Method)
  java.base@21/java.lang.Thread.getAllStackTraces(Thread.java:2521)
  app//nextflow.util.SysHelper.dumpThreads(SysHelper.groovy:188)
  app//nextflow.Session.abort(Session.groovy:800)
  app//nextflow.Session.fault(Session.groovy:766)
  app//nextflow.processor.TaskPollingMonitor.finalizeTask(TaskPollingMonitor.groovy:695)
  app//nextflow.processor.TaskPollingMonitor.safeFinalizeTask(TaskPollingMonitor.groovy:678)
  java.base@21/java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(DirectMethodHandle$Holder)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070011a8400.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:154)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  java.base@21/java.lang.reflect.Method.invoke(Method.java:580)
  app//org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
  app//groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
  app//org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:645)
  app//org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:628)
  app//org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:82)
  app//nextflow.processor.TaskPollingMonitor$_checkTaskStatus_lambda8.doCall(TaskPollingMonitor.groovy:668)
  app//nextflow.processor.TaskPollingMonitor$$Lambda/0x0000007001671e40.run(Unknown Source)
  java.base@21/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
  java.base@21/java.util.concurrent.FutureTask.run(FutureTask.java:317)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#54,AWSBatch-executor-11,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1758)
  java.base@21/java.util.concurrent.PriorityBlockingQueue.poll(PriorityBlockingQueue.java:549)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#9,Reference Handler,10,system]
  java.base@21/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  java.base@21/java.lang.ref.Reference.processPendingReferences(Reference.java:246)
  java.base@21/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:208)

Thread[#63,Actor Thread 9,5,main]
  app//com.esotericsoftware.kryo.util.DefaultClassResolver.<init>(DefaultClassResolver.java:21)
  app//com.esotericsoftware.kryo.Kryo.<init>(Kryo.java:132)
  java.base@21/java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(DirectMethodHandle$Holder)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x000000700101c000.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x000000700115dc00.guardWithCatch(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070010b2000.reinvoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070010b2400.guard(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070010b2000.reinvoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000007001138800.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
  app//org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
  java.base@21/java.lang.invoke.LambdaForm$DMH/0x00000070010aa800.invokeStatic(LambdaForm$DMH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070010b0000.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
  app//nextflow.util.KryoHelper.newInstance(SerializationHelper.groovy:87)
  java.base@21/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x000000700101c000.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:153)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  java.base@21/java.lang.reflect.Method.invoke(Method.java:580)
  app//org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
  app//groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
  app//groovy.lang.MetaClassImpl.invokeStaticMethod(MetaClassImpl.java:1586)
  app//org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:611)
  app//org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:182)
  app//nextflow.util.KryoHelper$1.methodMissing(SerializationHelper.groovy)
  java.base@21/java.lang.invoke.LambdaForm$DMH/0x00000070010b2c00.invokeVirtual(LambdaForm$DMH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070011ca000.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000007001182800.invokeExact_MT(LambdaForm$MH)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:155)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  java.base@21/java.lang.reflect.Method.invoke(Method.java:580)
  app//org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
  app//groovy.lang.MetaClassImpl.invokeMissingMethod(MetaClassImpl.java:924)
  app//groovy.lang.MetaClassImpl.invokePropertyOrMissing(MetaClassImpl.java:1413)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1335)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
  java.base@21/java.lang.invoke.LambdaForm$DMH/0x00000070010b3800.invokeInterface(LambdaForm$DMH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000007001179400.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x000000700119c800.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x000000700115dc00.guardWithCatch(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070010b2000.reinvoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070010b2400.guard(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070010b2000.reinvoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070010b2400.guard(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070010b2000.reinvoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000007001138800.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
  app//org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
  java.base@21/java.lang.invoke.LambdaForm$DMH/0x00000070010aa800.invokeStatic(LambdaForm$DMH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070010b0000.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
  app//nextflow.util.KryoHelper$1.initialValue(SerializationHelper.groovy:60)
  app//nextflow.util.KryoHelper$1.initialValue(SerializationHelper.groovy)
  java.base@21/java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:225)
  java.base@21/java.lang.ThreadLocal.get(ThreadLocal.java:194)
  java.base@21/java.lang.ThreadLocal.get(ThreadLocal.java:172)
  java.base@21/java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(DirectMethodHandle$Holder)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x000000700115dc00.guardWithCatch(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070010b2000.reinvoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070010b2400.guard(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070010b2000.reinvoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070010b2400.guard(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070010b2000.reinvoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000007001138800.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
  app//org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
  java.base@21/java.lang.invoke.LambdaForm$DMH/0x00000070010aa800.invokeStatic(LambdaForm$DMH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070010b0000.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
  app//nextflow.util.KryoHelper.kryo(SerializationHelper.groovy:79)
  java.base@21/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x000000700101c000.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x000000700115dc00.guardWithCatch(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070010b2000.reinvoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070010b2400.guard(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070010b2000.reinvoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000007001138800.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
  app//org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
  java.base@21/java.lang.invoke.LambdaForm$DMH/0x00000070010aa800.invokeStatic(LambdaForm$DMH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070010b0000.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
  app//nextflow.util.KryoHelper.serialize(SerializationHelper.groovy:166)
  app//nextflow.trace.TraceRecord.serialize(TraceRecord.groovy:573)
  app//nextflow.cache.CacheDB.writeTaskEntry0(CacheDB.groovy:147)
  java.base@21/java.lang.invoke.LambdaForm$DMH/0x00000070015af000.invokeVirtual(LambdaForm$DMH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x00000070016a9400.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000007001182800.invokeExact_MT(LambdaForm$MH)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:155)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  java.base@21/java.lang.reflect.Method.invoke(Method.java:580)
  app//org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
  app//groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
  app//org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:645)
  app//org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:628)
  app//org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:82)
  app//nextflow.cache.CacheDB$_putTaskAsync_closure1.doCall(CacheDB.groovy:157)
  app//nextflow.cache.CacheDB$_putTaskAsync_closure1.call(CacheDB.groovy)
  app//groovyx.gpars.agent.AgentBase.onMessage(AgentBase.java:102)
  app//groovyx.gpars.agent.Agent.handleMessage(Agent.java:84)
  app//groovyx.gpars.agent.AgentCore$1.handleMessage(AgentCore.java:48)
  app//groovyx.gpars.util.AsyncMessagingCore.run(AsyncMessagingCore.java:132)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#53,AWSBatch-executor-10,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1758)
  java.base@21/java.util.concurrent.PriorityBlockingQueue.poll(PriorityBlockingQueue.java:549)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#37,Actor Thread 2,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack$SNode.block(SynchronousQueue.java:288)
  java.base@21/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
  java.base@21/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:397)
  java.base@21/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:886)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#1,main,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519)
  java.base@21/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
  java.base@21/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1707)
  app//nextflow.util.Barrier$_awaitCompletion_closure3.doCall(Barrier.groovy:92)
  app//nextflow.util.Barrier$_awaitCompletion_closure3.call(Barrier.groovy)
  app//nextflow.extension.Bolts.withLock(Bolts.groovy:337)
  app//nextflow.extension.Bolts.withLock(Bolts.groovy)
  app//nextflow.util.Barrier.awaitCompletion(Barrier.groovy:89)
  app//nextflow.Session.await(Session.groovy:679)
  app//nextflow.script.ScriptRunner.await(ScriptRunner.groovy:256)
  app//nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:144)
  app//nextflow.cli.CmdRun.run(CmdRun.groovy:376)
  app//nextflow.cli.Launcher.run(Launcher.groovy:503)
  app//nextflow.cli.Launcher.main(Launcher.groovy:658)

Thread[#36,Actor Thread 1,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack$SNode.block(SynchronousQueue.java:288)
  java.base@21/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
  java.base@21/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:397)
  java.base@21/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:886)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#38,Actor Thread 3,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack$SNode.block(SynchronousQueue.java:288)
  java.base@21/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
  java.base@21/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:397)
  java.base@21/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:886)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#62,Actor Thread 8,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack$SNode.block(SynchronousQueue.java:288)
  java.base@21/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
  java.base@21/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:397)
  java.base@21/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:886)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#35,Task submitter,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519)
  java.base@21/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
  java.base@21/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1707)
  app//nextflow.processor.TaskPollingMonitor.awaitTasks(TaskPollingMonitor.groovy:388)
  app//nextflow.processor.TaskPollingMonitor.submitLoop(TaskPollingMonitor.groovy:412)
  java.base@21/java.lang.invoke.LambdaForm$DMH/0x00000070011ac000.invokeVirtual(LambdaForm$DMH)
  java.base@21/java.lang.invoke.LambdaForm$MH/0x0000007001520400.invoke(LambdaForm$MH)
  java.base@21/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:153)
  java.base@21/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  java.base@21/java.lang.reflect.Method.invoke(Method.java:580)
  app//org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
  app//groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethodClosure(MetaClassImpl.java:1017)
  app//groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1207)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
  app//groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
  app//groovy.lang.Closure.call(Closure.java:433)
  app//groovy.lang.Closure.call(Closure.java:412)
  app//groovy.lang.Closure.run(Closure.java:505)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#61,Actor Thread 7,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack$SNode.block(SynchronousQueue.java:288)
  java.base@21/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
  java.base@21/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
  java.base@21/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:397)
  java.base@21/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:886)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Thread[#56,AWSBatch-executor-13,5,main]
  java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@21/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
  java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1758)
  java.base@21/java.util.concurrent.PriorityBlockingQueue.poll(PriorityBlockingQueue.java:549)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
  java.base@21/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  java.base@21/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  java.base@21/java.lang.Thread.runWith(Thread.java:1596)
  java.base@21/java.lang.Thread.run(Thread.java:1583)

Jan-13 19:27:41.073 [TaskFinalizer-1] DEBUG nextflow.processor.TaskRun - Unable to dump error of process 'HelloWorld (hello)' -- Cause: java.nio.file.NoSuchFileException: /var/folders/3p/qh68nr3j5h3054n7llf_9l1r00025j/T/temp-s3-16858657256812211440/.command.err
Jan-13 19:27:41.290 [TaskFinalizer-1] DEBUG nextflow.processor.TaskRun - Unable to dump output of process 'HelloWorld (hello)' -- Cause: java.nio.file.NoSuchFileException: /var/folders/3p/qh68nr3j5h3054n7llf_9l1r00025j/T/temp-s3-3699814231108743939/.command.out
Jan-13 19:27:41.291 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jan-13 19:27:41.292 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: awsbatch) - terminating tasks monitor poll loop
Jan-13 19:27:41.487 [main] DEBUG nextflow.processor.TaskRun - Unable to dump error of process 'HelloWorld (hello)' -- Cause: java.nio.file.NoSuchFileException: /var/folders/3p/qh68nr3j5h3054n7llf_9l1r00025j/T/temp-s3-17343225520198514509/.command.err
Jan-13 19:27:41.678 [main] DEBUG nextflow.processor.TaskRun - Unable to dump output of process 'HelloWorld (hello)' -- Cause: java.nio.file.NoSuchFileException: /var/folders/3p/qh68nr3j5h3054n7llf_9l1r00025j/T/temp-s3-13491829741892424922/.command.out
Jan-13 19:27:41.688 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=1; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=0ms; failedDuration=2s; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=1; peakMemory=8 GB; ]
Jan-13 19:27:41.696 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Jan-13 19:27:41.723 [main] INFO  org.pf4j.AbstractPluginManager - Stop plugin '[email protected]'
Jan-13 19:27:41.723 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-amazon
Jan-13 19:27:41.725 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

pditommaso Jan 13, 2025
Maintainer

To tell the truth dont know what the problem is. Wave is not involved in this run. The container looks working properly

docker run community.wave.seqera.io/library/bcftools_pip_awscli:cae216d906baa276 bash -c 'echo hello'
hello

Shians Jan 14, 2025
Author

Thanks for helping out, does "/usr/local/bin/_entrypoint.sh" sound like something that Nextflow or the Seqera image creates or something from Docker? The main issue with my attempts to debug this is that I have no idea what Nextflow, the AWS plugin, the Seqera image or Docker on the remote are doing, so I have no idea what stage of the process is actually causing a break.

I know that using container 'amazonlinux:2023' does work, but it doesn't have the software I need. So do you have any recommendations for how I would use that image while also having something like bcftools? Trying to use Nextflow's Conda directive doesn't seem to work because as you mentioned Conda runs locally while I need the package remotely. Do I just include the whole installation process into the process script? That feels a bit unclean to me.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"CannotStartContainerError: Error response from daemon: failed to create task for container" when using seqera containers #5659

{{title}}

Replies: 1 comment 9 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

"CannotStartContainerError: Error response from daemon: failed to create task for container" when using seqera containers #5659

Shians Jan 9, 2025

Bug report

Steps to reproduce the problem

Program output

Environment

Replies: 1 comment · 9 replies

pditommaso Jan 9, 2025 Maintainer

Shians Jan 13, 2025 Author

pditommaso Jan 13, 2025 Maintainer

Shians Jan 13, 2025 Author

pditommaso Jan 13, 2025 Maintainer

Shians Jan 14, 2025 Author

Shians
Jan 9, 2025

Replies: 1 comment 9 replies

pditommaso
Jan 9, 2025
Maintainer

Shians Jan 13, 2025
Author

pditommaso Jan 13, 2025
Maintainer

Shians Jan 13, 2025
Author

pditommaso Jan 13, 2025
Maintainer

Shians Jan 14, 2025
Author