Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to stdin to cp #182

Closed
mrekim0 opened this issue Jul 2, 2020 · 7 comments · Fixed by #587
Closed

Add option to stdin to cp #182

mrekim0 opened this issue Jul 2, 2020 · 7 comments · Fixed by #587

Comments

@mrekim0
Copy link

mrekim0 commented Jul 2, 2020

Basically the other end of #20, which resulted in s5cmd cat. It doesn't look like using s5cmd run or cp has the magic "-" option to take in a piped input from stdin.

Use cases: Pipe a tar archive (or any streamed command output) into s5cmd without needing a intermediary temp file. This also solves cases where we want to preserve timestamps, permissions, etc.

Example:
tar -cf - foo | s5cmd cp - s3://bar/foo.tar

or if using s5cmd run

tar -cf - foo | echo "cp - s3://bar/foo.tar" | s5cmd run

@onlined
Copy link
Contributor

onlined commented Jul 26, 2020

Second example passes bytes of tar archive to echo which ignores standard input and just passes cp - s3://bar//foo.tar to s5cmd run's standard input. I think this is not the intended behavior.

@nhoughto
Copy link

Can't support for - just be added without removing cat? taking bytes from stdin would be very handy for use in lambdas.

@david-macleod
Copy link

Agree that this would be very useful

@mlissner
Copy link

I'm looking to use s5cmd to do database backups. Postgres can stream to stdout. Is there a trick for using s5cmd to stream that to s3, so I don't have to put it on disk first? I know s5cmd has been around awhile, so perhaps there's some CMD trick folks are using to work around this?

@stefanoabbate
Copy link

It would be nice to have a "pipe" command similar to https://docs.min.io/docs/minio-client-complete-guide.html#pipe
s5cmd is terribly faster so we switched to this tool!

@suryatmodulus
Copy link

@igungor Can we achive an implementation similar to this ? https://rclone.org/commands/rclone_rcat/

@sfxworks
Copy link

sfxworks commented May 21, 2023

Would love this feature. Just tried and it doesn't work.
dd if=/dev/nvme0n1 bs=4M | pv | s5cmd cp - s3://home/backup/pc.img
I don't have room to even compress this nvme locally

@igungor igungor added this to the v2.2.0 milestone Jun 19, 2023
@ilkinulas ilkinulas added this to s5cmd Jul 3, 2023
ahmethakanbesel added a commit to ahmethakanbesel/s5cmd that referenced this issue Jul 6, 2023
@ahmethakanbesel ahmethakanbesel moved this to In Progress in s5cmd Jul 10, 2023
@ilkinulas ilkinulas moved this from In Progress to Review in s5cmd Jul 20, 2023
seruman pushed a commit that referenced this issue Jul 27, 2023
@github-project-automation github-project-automation bot moved this from Review to Done in s5cmd Jul 27, 2023
igungor added a commit that referenced this issue Jul 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
10 participants