Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use cp -c when copying files #17373

Merged
merged 19 commits into from
Jun 13, 2024
Merged

Use cp -c when copying files #17373

merged 19 commits into from
Jun 13, 2024

Conversation

tesaguri
Copy link
Contributor

@tesaguri tesaguri commented May 27, 2024

  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same change?
  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your changes? Here's an example.
  • Have you successfully run brew style with your changes locally?
  • Have you successfully run brew typecheck with your changes locally?
  • Have you successfully run brew tests with your changes locally?

This patch makes Homebrew use cp --reflink=auto if coreutils formula is installed, which significantly speeds up build/installation operations and reduces the risk of exhausting the storage during operations especially when dealing with large formulae.

I will work on new tests if the idea sounds good to you.

tesaguri added 2 commits May 27, 2024 12:11
This module determines the `cp` command to use based on availability of
the `coreutils` formula and optimizes the command invocation to prefer a
lightweight copy-on-write clone, which is significantly faster than a
full file copy and helps to reduce the risk of exhausting the storage
during the operation.
This replaces `FileUtils.cp` and `system_command! "cp"` with the new
`Utils::Cp` utility where it is expected that the performance
improvement outweighs the cost of the system command invocation.
@Bo98
Copy link
Member

Bo98 commented May 27, 2024

There is a cost of shelling out that should be considered here too. FileUtils.cp does not shell out. You should present your case with numbers for each of the call sites changed with both small and large payloads.

FWIW what happens under the hood with reflink is it calls clonefile on macOS and does a ioctl(FICLONE, ...) on Linux, though I guess there isn't Ruby bindings for those.

macOS's default cp supports a similar feature with the -c flag since I think macOS 10.13, so that seems like a much easier option for the cask code that already shells out.

@tesaguri
Copy link
Contributor Author

There is a cost of shelling out that should be considered here too. FileUtils.cp does not shell out. You should present your case with numbers for each of the call sites changed with both small and large payloads.

I have tried to limit the use of the system command to places that I guessed to be likely to deal with large payloads, but yes, that's only wild guess and I surely need some benchmarks.

macOS's default cp supports a similar feature with the -c flag since I think macOS 10.13, so that seems like a much easier option for the cask code that already shells out.

I have considered that too, but I didn't adopt it because it fails if the clonefile syscall isn't applicable to the source and destination of the copy, as I wrote in https://github.com/Homebrew/brew/pull/17373/files#diff-52b1a61a5f4b7c031445c5f76c206db7c60089cb7204d7eeacbef0a0924e8553R61-R64.

(And sorry for the test failure. I forgot to run the test after rebasing on latest master. I'll fix it.)

@Bo98
Copy link
Member

Bo98 commented May 27, 2024

have considered that too, but I didn't adopt it because it fails if the clonefile syscall isn't applicable to the source and destination of the copy, as I wrote in https://github.com/Homebrew/brew/pull/17373/files#diff-52b1a61a5f4b7c031445c5f76c206db7c60089cb7204d7eeacbef0a0924e8553R61-R64.

Ok fair point, though it does also seem macOS 14 shipped with a change to -c:

     -c    copy files using clonefile(2).  Note that if clonefile(2) is not
           supported for the target filesystem, then cp will fallback to using
           copyfile(2) instead to ensure the copy still succeeds.

Under the hood, it does this by checking if clonefile returns ENOTSUP. Are you still seeing the behaviour you describe on macOS 14?

@tesaguri
Copy link
Contributor Author

Are you still seeing the behaviour you describe on macOS 14?

No. I was just writing in terms of the compatibility with old systems and it's not that I observe the behavior on the latest system.

Perhaps I can make the module change the strategy based on macOS version.

@tesaguri tesaguri changed the title Cp reflink Use cp --reflink=auto when copying files May 27, 2024
@Bo98
Copy link
Member

Bo98 commented May 27, 2024

Even gating the whole feature to >= macOS 14 is probably ok. It would likely significantly simplify the code and that seems acceptable given we've gone on fine without the feature this long and older macOS versions cover only ~15% of users, and that figure will only decrease further (macOS 15 is announced next month).

@MikeMcQuaid
Copy link
Member

Thanks for the PR @tesaguri!

Even gating the whole feature to >= macOS 14 is probably ok.

Yes, this seems like the right approach.

You should present your case with numbers for each of the call sites changed with both small and large payloads.

Not just microbenchmarks either but the benchmarks of actually installing e.g. a relevant formula/cache.

Do this before adding/changing any tests as if the performance gain is not significant this won't be accepted.

@tesaguri tesaguri changed the title Use cp --reflink=auto when copying files Use cp -c when copying files Jun 6, 2024
Copy link
Member

@MikeMcQuaid MikeMcQuaid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @tesaguri, looking great and we're almost there! ❤️

@tesaguri
Copy link
Contributor Author

tesaguri commented Jun 7, 2024

I've done a quick benchmark of the execution time of brew [un]install with a cask (.dmg) and a bottled formula (.tar.gz).

The following tables are output of the hyperfine benchmarking tool and the Command names master-* and reflink-* represent the results of Homebrew:master branch and tesaguri:cp-reflink branch (this patch) respectively.

brew install libreoffice:

Command Mean [s] Min [s] Max [s] Relative
master-install-libreoffice 84.196 ± 6.290 78.934 100.503 1.41 ± 0.13
reflink-install-libreoffice 59.804 ± 2.992 57.261 67.991 1.00

brew uninstall libreoffice:

Command Mean [s] Min [s] Max [s] Relative
master-uninstall-libreoffice 45.217 ± 2.183 41.470 49.262 3.17 ± 0.24
reflink-uninstall-libreoffice 14.274 ± 0.844 12.609 15.150 1.00

brew install llvm:

Command Mean [s] Min [s] Max [s] Relative
master-install-llvm 137.381 ± 4.537 132.001 148.779 1.29 ± 0.05
reflink-install-llvm 106.714 ± 1.671 104.831 109.686 1.00

brew uninstall llvm:

Command Mean [s] Min [s] Max [s] Relative
master-uninstall-llvm 9.514 ± 0.782 8.597 10.871 1.00
reflink-uninstall-llvm 9.684 ± 0.846 8.459 11.047 1.02 ± 0.12

(The uninstallation seems to be unaffected by the change, as expected.)

Also, I've tested with the bottle file as the argument just to be sure.

brew install "$(brew --cache llvm)":

Command Mean [s] Min [s] Max [s] Relative
master-install-llvm_bottle 148.959 ± 2.346 146.004 152.294 1.15 ± 0.02
reflink-install-llvm_bottle 129.275 ± 1.413 127.639 132.201 1.00

So far, the results are easily expected given the large size of these cask and formula (and there are no additional command invocation cost for casks anyway).

But the pessimistic case would be small formulae for which the cost of system command invoctions relative to the IO cost is expected to be higher.

To simulate the most pessimistic case, I've built a bottle with only a single zero-sized file and measured the installation time of it (the source of the formula will be shown later).

Because the bottle is so small that it is hard to calibrate the time taken by unrelated tasks for sure, I've created 100 such bottles and installed them at once in each brew command invocation.

brew install -- test-formula-tiny*--0.0.0.all.bottle.tar.gz:

Command Mean [s] Min [s] Max [s] Relative
master-install-test-formula-tiny000_bottle.test-formula-tiny001_bottle.test-formula-tiny002_bottle 31.911 ± 0.579 31.014 33.254 1.21 ± 0.03
reflink-install-test-formula-tiny000_bottle.test-formula-tiny001_bottle.test-formula-tiny002_bottle 26.359 ± 0.294 25.876 26.873 1.00

brew uninstall -- test-formula-tiny*--0.0.0.all.bottle.tar.gz:

Command Mean [s] Min [s] Max [s] Relative
master-uninstall-test-formula-tiny000_bottle.test-formula-tiny001_bottle.test-formula-tiny002_bottle 18.868 ± 0.475 18.263 19.584 1.80 ± 0.06
reflink-uninstall-test-formula-tiny000_bottle.test-formula-tiny001_bottle.test-formula-tiny002_bottle 10.505 ± 0.258 10.099 10.975 1.00

The improvement in the uninstallation time is surprising to me. I think I'm missing something… Perhaps, this has something to do with the direct uninstallation from bottle files, which involves extraction of the bottles, I guess?

Anyway, it seems that the change improves the performance even in the pessimistic case (at least it doesn't degrade the performance).

(I haven't come up with test cases of other download strategies like Xz that represent realistic use cases well enough. Also, I couldn't figure out any code paths that lead to the cp invocation in Pathname#cp_sub_path method, because it only calls mkpath if self.directory? is true and all of its callers seem to call the method only when directory? is true.)

Reproduction

Source code of the test formula:

class TestFormulaTiny000 < Formula
  url "file:///dev/null"
  version "0.0.0"
  sha256 "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"

  def install
    mkdir_p "test-tiny"
    mv "null", "test-tiny/000"
    share.install "test-tiny"
  end
end

Duplicate it and build them with the following command:

$ for i in {001..099}; do sed "s/000/$i/g" test-formula-tiny000.rb > "test-formula-tiny$i.rb"; done
$ brew install --build-bottle -- test-formula-tiny{000..099}

Here is a helper script for the benchmark (requires hyperfine formula to be installed):

bench.sh
#!/bin/bash

set -e

# WORKTREE=(path to the worktree of the fork)
WORKTREE_BREW="$WORKTREE/bin/brew"

HYPERFINE_FLAGS='--warmup=3'
BREW_INSTALL_FLAGS='--ignore-dependencies --skip-cask-deps'
BREW_UNINSTALL_FLAGS='--ignore-dependencies'

cd "$(dirname "$0")"

subcommand="$1"
shift

args=()
pkgs=()
for arg in "$@"; do
	args+=("$(printf '%q' "$arg")")
	if ! [[ "$arg" = -* ]]; then
		if [[ "$arg" = *.bottle.tar.gz ]]; then
			arg="${arg##*/}"
			arg="${arg%.bottle.tar.gz}"
			arg="${arg%--*}"
			arg="${arg#*--}_bottle"
		fi
		pkgs+=("$arg")
	fi
done
pkgs_label="$(IFS='.'; echo "${pkgs[*]:0:3}")"
pkgs_label="${pkgs_label:0:240}"

for env in $(export | grep '^HOMEBREW_'); do
	unset "$env"
done

export HOMEBREW_CURL_RETRIES=0
export HOMEBREW_NO_AUTOREMOVE=1
export HOMEBREW_NO_AUTO_UPDATE=1
export HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK=1
export HOMEBREW_NO_INSTALL_CLEANUP=1
export HOMEBREW_NO_INSTALL_UPGRADE=1

case "$subcommand" in
install)
	fetch=()
	for arg in "${args[@]}"; do
		case "$arg" in
		-*|*.bottle.tar.gz)
			;;
		*)
			fetch+=("$arg")
			;;
		esac
	done
	mkdir -p results
	hyperfine $HYPERFINE_FLAGS --shell=none \
		--export-json="results/install-$pkgs_label.json" \
		--export-markdown="results/install-$pkgs_label.md" \
		${fetch:+--setup="sh -c 'brew fetch ${fetch[*]} || true'"} \
		--command-name="master-install-$pkgs_label" \
		--prepare="brew uninstall --force $BREW_UNINSTALL_FLAGS ${args[*]}" \
		"brew install $BREW_INSTALL_FLAGS ${args[*]}" \
		--command-name="reflink-install-$pkgs_label" \
		--prepare="$(printf '%q' "$WORKTREE_BREW") uninstall --force $BREW_UNINSTALL_FLAGS ${args[*]}" \
		"$(printf '%q' "$WORKTREE_BREW") install $BREW_INSTALL_FLAGS ${args[*]}"
	;;
uninstall)
	mkdir -p results
	hyperfine $HYPERFINE_FLAGS --shell=none \
		--export-json="results/uninstall-$pkgs_label.json" \
		--export-markdown="results/uninstall-$pkgs_label.md" \
		--command-name="master-uninstall-$pkgs_label" \
		--prepare="brew install --force $BREW_INSTALL_FLAGS ${args[*]}" \
		"brew uninstall $BREW_UNINSTALL_FLAGS ${args[*]}" \
		--command-name="reflink-uninstall-$pkgs_label" \
		--prepare="$(printf '%q' "$WORKTREE_BREW") install --force $BREW_INSTALL_FLAGS ${args[*]}" \
		"$(printf '%q' "$WORKTREE_BREW") uninstall $BREW_UNINSTALL_FLAGS ${args[*]}"
	;;
*)
	echo "Usage: ${0##*/} install|uninstall PACKAGE..."
	exit 1
	;;
esac

Run the benchmark with the following commands:

$ brew fetch -- libreoffice llvm
$ : Here, I have disconnected my machine from the internet to make `curl` fail immidiately,
$ : reducing the variance due to the network condition.
$ export WORKTREE=(path to the worktree of the fork); \
	caffeinate ./bench.sh install -- libreoffice; \
	caffeinate ./bench.sh uninstall -- libreoffice; \
	caffeinate ./bench.sh install -- llvm; \
	caffeinate ./bench.sh uninstall -- llvm; \
	caffeinate ./bench.sh install -- "$(brew --cache -- llvm)"; \
	caffeinate ./bench.sh install -- test-formula-tiny{000..099}--0.0.0.all.bottle.tar.gz; \
	caffeinate ./bench.sh uninstall -- test-formula-tiny{000..099}--0.0.0.all.bottle.tar.gz

Copy link
Member

@MikeMcQuaid MikeMcQuaid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some more naming thoughts. This is very nearly ready to 🚢! Very pleased by performance increases!

@MikeMcQuaid
Copy link
Member

Thanks for the detailed write-up @tesaguri! Are you using hyperfine for these benchmarks? If so, use the --warmup option if you're not already as it may help explain some of the unexpected results.

@tesaguri
Copy link
Contributor Author

tesaguri commented Jun 8, 2024

Are you using hyperfine for these benchmarks? If so, use the --warmup option if you're not already as it may help explain some of the unexpected results.

Yes, I'm already using --warmup=3.

@Bo98
Copy link
Member

Bo98 commented Jun 8, 2024

This is pretty cool and has inspired me to look into adding clonefile to Ruby upstream. The Linux equivalent appears to already be implemented so seems reasonable to bring macOS in line.

`FileUtils.cp` is implemented with the lightweight `copy_file_range(2)`
syscall on Linux, so it's more performant than the plain `cp` command on
that platform.

cf. Homebrew#17373 (review)
@tesaguri
Copy link
Contributor Author

tesaguri commented Jun 8, 2024

https://github.com/Homebrew/brew/actions/runs/9428441435/job/25973762028?pr=17373#step:12:42

Well, I have forgotten that UnpackStrategy::Directory is also using the cp command.

https://github.com/Homebrew/brew/blob/master/Library/Homebrew/unpack_strategy/directory.rb#L23-L26

Reverting this code to master should fix the test, but I think that UnpackStrategy::Directory is one of the most performance-critical parts and that simply reverting this change spoils the point of the patch.

Perhaps, we can add yet another utility method (Edit: or a new keyword argument for existing methods?) that is guaranteed to use cp command, though I expect that would increase code complexity.

Another option is to revert the latest commit (9156891), but that would lead to a significant performance regression on Linux.

Or should we abandon the whole patch and wait for Ruby upstream to use clonefile (if ever)? But UnpackStrategy::Directory and Cask::Artifact::Moved (which uses cp command) would be stuck in the status-quo performance that way.

This fixes the test for `UnpackStrategy::Directory`, which needs the
`cp` command.
@tesaguri
Copy link
Contributor Author

tesaguri commented Jun 9, 2024

Added force_command (the name is up for bikeshedding) keyword arguments to force the use of the shell command in 9156891, which I think ended up reasonably simple (compared to the original idea of adding dedicated methods instead).

@Bo98
Copy link
Member

Bo98 commented Jun 9, 2024

I gave it a go: Bo98/ruby@6fc57a2.

$ mkfile -n 1g big

$ hyperfine --warmup 3 --prepare "rm big2 || true" 'ruby-orig -e "IO.copy_stream(\"big\", \"big2\")"'
Benchmark 1: ruby-orig -e "IO.copy_stream(\"big\", \"big2\")"
  Time (mean ± σ):     251.1 ms ±   8.3 ms    [User: 23.1 ms, System: 141.8 ms]
  Range (min … max):   238.6 ms … 264.5 ms    11 runs

$ hyperfine --warmup 3 --prepare "rm big2 || true" 'ruby-new -e "IO.copy_stream(\"big\", \"big2\")"'
Benchmark 1: ruby-new -e "IO.copy_stream(\"big\", \"big2\")"
  Time (mean ± σ):      25.9 ms ±   0.9 ms    [User: 21.5 ms, System: 3.6 ms]
  Range (min … max):    25.2 ms …  33.7 ms    97 runs

Works on macOS 10.13+.
I'll discuss this with upstream. If it is accepted, I will backport this to our Ruby. One catch is that in order for it to take effect on Intel, we would need to bump the minimum macOS to 10.12 (arm64 already has a macOS 11 minimum so that's ok). There's graceful fallback on 10.12, but not for 10.11. This doesn't necessarily need to happen immediately - we can backport and the configure checks will automatically only apply the new code to arm64 until we bump the minimum.

Or should we abandon the whole patch and wait for Ruby upstream to use clonefile (if ever)?

I am ok with this PR as long as the commits are structured in a way that it's easy to revert the unpack_strategy changes if Ruby itself gets patched.

Casks will still use shelling out due to sudo needs, and those are also macOS only.

@Bo98
Copy link
Member

Bo98 commented Jun 9, 2024

Well, I have forgotten that UnpackStrategy::Directory is also using the cp command.

Hmm that's annoying. Really it should use FileUtils, but it seems it doesn't for some reason. Will check and see if that reason is still valid.

Copy link
Member

@MikeMcQuaid MikeMcQuaid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more question/discussion on some names and I think we're good to go here, thanks @tesaguri!

@tesaguri
Copy link
Contributor Author

I've added tests for the new module in 4edbbfd.

I'm going to edit the commit history after the review completes, so that it's easier to partially revert the change if/when FileUtils supports clonefile(2).

Copy link
Member

@MikeMcQuaid MikeMcQuaid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work @tesaguri! One more comment and then I'm ✅, thanks again for such great work here!

@MikeMcQuaid
Copy link
Member

MikeMcQuaid commented Jun 13, 2024

Thanks so much for your first contribution! Without people like you submitting PRs we couldn't run this project. You rock, @tesaguri; this is one of the highest quality first PRs I've ever seen in 15 years maintaining Homebrew! I hope to see many more from you ❤️

@MikeMcQuaid MikeMcQuaid merged commit 1e0add6 into Homebrew:master Jun 13, 2024
25 checks passed
@apainintheneck
Copy link
Contributor

It looks like this change is causing some problems for folks so I opened a revert PR.

@tesaguri
Copy link
Contributor Author

Ah, sorry for the regression. Is the revert intended to be temporary?

If so, it looks like the issues could be fixed by using fully-qualified path for the cp command, i.e., "/bin/cp", though I haven't closely looked into the issues yet and it would definitely need some sort of regression tests.

@apainintheneck
Copy link
Contributor

Ah, sorry for the regression. Is the revert intended to be temporary?

No worries. It happens. We're usually open to retrying PRs that get reverted as long as the original issues get addressed. I'm just not super familiar with this code personally so I didn't see an obvious way to fix forward last night. Feel free to open another PR for that.

@MikeMcQuaid
Copy link
Member

it looks like the issues could be fixed by using fully-qualified path for the cp command, i.e., "/bin/cp",

This sounds great, thanks!

@jmcbsimmonds
Copy link

Hello,

It seems adding "-c" option to cp can cause an error on MacOS Sonoma 14.9:

For example:
Error: zoom: Failure while executing; /usr/bin/env cp -p -c /Volumes/Home/Users/username/Library/Caches/Homebrew/downloads/960197a91564aadc18ea997951366349d9a9604c79f350db9b259cdc9000f7e2--zoomusInstallerFull.pkg /usr/local/Caskroom/zoom/6.1.0.35886/zoomusInstallerFull.pkg exited with 1. Here's the output:
cp: /usr/local/Caskroom/zoom/6.1.0.35886/zoomusInstallerFull.pkg: clonefile failed: Cross-device link

The / filesystem is APFS, the /Volumes/Home filesystem is APFS (Case Sensitive)

Same error with Microsoft-Edge.

HOMEBREW_VERSION: 4.3.6
ORIGIN: https://github.com/Homebrew/brew
HEAD: e8430b2
Last commit: 6 days ago
Core tap JSON: 22 Jun 05:42 UTC
Core cask tap JSON: 22 Jun 05:42 UTC
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 16
Homebrew Ruby: 3.3.3 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.3.3/bin/ruby
CPU: 16-core 64-bit kabylake
Clang: 15.0.0 build 1500
Git: 2.45.2 => /usr/local/bin/git
Curl: 8.6.0 => /usr/bin/curl
macOS: 14.5-x86_64
CLT: 15.3.0.0.1.1708646388
Xcode: 15.4

Is there any config option to turn off -c option?

Kind regards.

@apainintheneck
Copy link
Contributor

@jmcbsimmonds This change has been reverted but has not made it into the most recent release yet. The weekly release usually comes out Monday-ish.

@MikeMcQuaid
Copy link
Member

This is released/fixed in Homebrew 4.3.7 now: https://github.com/Homebrew/brew/releases/tag/4.3.7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants