Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for enabling cloudcache from CLI #4385

Merged
merged 3 commits into from
Oct 9, 2023
Merged

Add support for enabling cloudcache from CLI #4385

merged 3 commits into from
Oct 9, 2023

Conversation

pditommaso
Copy link
Member

This adds the support for enabling cloudcache plugin via run CLI option -cloudcache [path]

@netlify
Copy link

netlify bot commented Oct 8, 2023 •
edited
Loading

✅ Deploy Preview for nextflow-docs-staging ready!

Name Link
🔨 Latest commit 8247c2c
🔍 Latest deploy log https://app.netlify.com/sites/nextflow-docs-staging/deploys/65246f796fe6bc00089477f6
😎 Deploy Preview https://deploy-preview-4385--nextflow-docs-staging.netlify.app/cli
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@pditommaso pditommaso requested a review from bentsherman October 8, 2023 16:02
@pditommaso
Copy link
Member Author

Too bad

Oct-08 16:33:19.230 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.CloudCacheFactory
Oct-08 16:33:19.231 [main] ERROR nextflow.cli.Launcher - NXF_CLOUDCACHE_PATH must be defined when using the cloud cache store
java.lang.IllegalArgumentException: NXF_CLOUDCACHE_PATH must be defined when using the cloud cache store
	at nextflow.cache.CloudCacheFactory.newInstance(CloudCacheFactory.groovy:44)
	at nextflow.cache.CacheFactory.create(CacheFactory.groovy:44)
	at nextflow.cache.CacheFactory.create(CacheFactory.groovy)
	at nextflow.Session.init(Session.groovy:420)
	at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:128)
	at nextflow.cli.CmdRun.run(CmdRun.groovy:372)
	at nextflow.cli.Launcher.run(Launcher.groovy:499)
	at nextflow.cli.Launcher.main(Launcher.groovy:670)

@pditommaso
Copy link
Member Author

@bentsherman any chance you can have a look why it's failing?

@pditommaso
Copy link
Member Author

Think i've found the problem

@pditommaso pditommaso removed the request for review from bentsherman October 9, 2023 18:42
@pditommaso
Fix tests
41c7a52 
Signed-off-by: Paolo Di Tommaso <[email protected]>
@pditommaso
Add CLI docs [ci skip]
8247c2c 
Signed-off-by: Paolo Di Tommaso <[email protected]>
@pditommaso pditommaso merged commit 73fda58 into master Oct 9, 2023
@pditommaso pditommaso deleted the add-cloudcache-cli branch October 9, 2023 21:31
abhi18av pushed a commit to abhi18av/nextflow that referenced this pull request Oct 28, 2023
@pditommaso @abhi18av
Add -cloudcache CLI option (nextflow-io#4385)
dd261bc 
This commit adds the -cloudcache run command line option that
allows storing Nextflow cache metadata into a cloud object storage
bucket instead of storing them in the local .nextflow directory.

The target cloud cache path is specified as command line argument,
e.g. -cloudcache s3://bucket-name/path.

Moreover, this commit allows controlling the cloudcache via the
nextflow.config file as shown below:

```
cloudcache.enabled = true|false
cloudcache.path = '<some object storage path>'
```

Supported object storage are AWS S3, Azure Blob storage and Google Storage.

Signed-off-by: Paolo Di Tommaso <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@pditommaso