create: implement --paths-from-stdin and --paths-from-command

[Lapin0t]

WIP. Follow-up of #5492. Implement two switches which shortcut the filesystem-walking logic completely: borg will read the paths and backup without doing any recursion (directories can --and should-- be passed). Should honor any relevant option (common options and all filesystem option but --one-file-system).

Slight change from my earlier proposal: now if either --paths-from-stdin or --paths-from-command are given, then it won't process any normal path from args.path. In --path-from-command it wouldn't make sense anyway since args.path is being used for the command. Technically, for --paths-from-stdin we could still handle args.path as usual but it's probably safe to assume that a user don't want to mix both things. Besides it simplifies the code and the top-level behavior of create.

WIP: Just wrote, didn't execute! :) I should verify if it works correctly and add unit tests to the PR.

[ThomasWaldmann]

src/borg/archiver.py:548:22: E261 at least two spaces before inline comment
src/borg/helpers/misc.py:217:1: E302 expected 2 blank lines, found 1

after you fix that, the real tests will run.

[ThomasWaldmann]

--path-from-command <-- there seems to be an "s" missing.

can you fix the commit comment, the PR top post, etc., please?

[ThomasWaldmann]

thanks for the PR, nice powerful functionality!

[ThomasWaldmann]

would be cool to get this into 1.2.0b1 (soon).

[Lapin0t]

Sorry for the force-push, i needed to edit the commit message...

[codecov-io]

Codecov Report

Merging #5538 (fcbb43d) into master (9fa28df) will increase coverage by 0.04%.
The diff coverage is 77.35%.

@@            Coverage Diff             @@
##           master    #5538      +/-   ##
==========================================
+ Coverage   83.08%   83.13%   +0.04%     
==========================================
  Files          38       38              
  Lines       10003    10054      +51     
  Branches     1657     1668      +11     
==========================================
+ Hits         8311     8358      +47     
- Misses       1202     1203       +1     
- Partials      490      493       +3     

Impacted Files	Coverage Δ
src/borg/archiver.py	80.25% <69.23%> (-0.14%)	⬇️
src/borg/helpers/misc.py	80.62% <100.00%> (+1.57%)	⬆️
src/borg/helpers/parseformat.py	89.88% <100.00%> (+0.03%)	⬆️
src/borg/archive.py	81.44% <0.00%> (+0.26%)	⬆️
src/borg/helpers/fs.py	86.66% <0.00%> (+1.33%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fa331c3...fcbb43d. Read the comment docs.

[ThomasWaldmann]

No problem, there is no other way than to force-push if one edits history.

Shall I merge it like this for now and we check the bytes vs. text paths later?

[Lapin0t]

No wait a bit, i'm testing it and there is a weird bug, it's really not doing what it should.

[ThomasWaldmann]

guess a test would be fine also.

[Lapin0t]

Ok, this should be fine now. The tests are mostly copy&paste from content-from-command.. not really exhaustive in any way neither. About the text/binary thing, i realized proc.stdout was already in binary mode by default, so i just changed sys.stdin.buffer and wrapped the whole thing in textmode with surrogateescape which seems to be the right thing to do when handling paths (i saw there are some helpers already doing that kind of encoding so maybe you know more about it).

[ThomasWaldmann]

Thanks! Merged!

I created a related issue #5551.