Skip to content

Commit

Permalink
Use an argsfile when calling merge_index_store.py
Browse files Browse the repository at this point in the history
Summary:
## Problem

Previously, larger buck targets (those with several thousand transitive dependencies) would fail when building the `full-index-store`, because the CLI invocation would be too big and hit Linux limits (around [128KiB](https://unix.stackexchange.com/questions/110282/cp-max-source-files-number-arguments-for-copy-utility/)).

This would produce build failures like this:

```
$ buck build fbobjc/Apps/Instagram/Features/Sundial/IGSundial:IGSundial[full-index-store]
Action failed: fbsource//fbobjc/Apps/Instagram/Features/Sundial/IGSundial:IGSundial (merge_index_store __indexstore__/IGSundial/full_index_stores)
Local command returned non-zero exit code <no exit code>
Reproduce locally: `env -- 'BUCK_SCRATCH_PATH=buck-out/v2/tmp/fbsource/6e8c89e8087abbf6/merge_index_store/__indexstore__ ...<omitted>... ource/33ed8d67ac42318a/fbobjc/VendorLib/IGListKit/__IGListKit__/__indexstore__/IGListKit/index_store (run `buck2 log what-failed` to get the full command)`
```

See P1746393724 for an example output from `buck log what-failed`, it's clearly a ton of data.

## Solution

Instead of invoking `merge_index_store.py --source dir1 dir2 dir3 ...`, use an arguments file as `merge_index_store.py --source directories.txt` so the CLI invocation is short.

This is a pattern used in other buck rules, so I based this code on these examples:

https://www.internalfb.com/code/fbsource/[834112476033]/fbcode/buck2/prelude/cxx/symbols.bzl?lines=293

https://www.internalfb.com/code/fbsource/[fa0f657e098379489efb65a65b20ce0d269fb276][history]/fbcode/buck2/prelude/apple/swift/swift_compilation.bzl?lines=690

Differential Revision: D70747622

fbshipit-source-id: 4f7f187e3e01352ca2da687b46591dc8601335a8
  • Loading branch information
Wilfred authored and facebook-github-bot committed Mar 7, 2025
1 parent 9535fab commit ddecda9
Show file tree
Hide file tree
Showing 2 changed files with 20 additions and 4 deletions.
10 changes: 9 additions & 1 deletion prelude/apple/tools/index/merge_index_store.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,15 @@ def merge_directories(source: str, destination: str) -> None:
def main() -> None:
args = parse_arguments()
destination = args.dest
directories = args.sources

directories = []
for source in args.sources:
if source.startswith("@"):
with open(source[1:]) as f:
for line in f.readlines():
directories.append(line.strip())
else:
directories.append(source)

Path(destination).mkdir(parents=True, exist_ok=True)

Expand Down
14 changes: 11 additions & 3 deletions prelude/cxx/index_store.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,19 @@ def _merge_index_store(ctx: AnalysisContext, index_stores: list[Artifact] | Tran

cmd = cmd_args([merge_index_store_tool])
cmd.add(["--dest", merged_index_store.as_output()])
cmd.add(["--sources"])

txt_args = cmd_args()
if isinstance(index_stores, list):
cmd.add(index_stores)
txt_args.add(index_stores)
else:
cmd.add(index_stores.project_as_args("args"))
txt_args.add(index_stores.project_as_args("args"))

argsfile_path = merge_output_dir_name + ".files.txt"
argsfile = ctx.actions.write(argsfile_path, txt_args)
argsfile = cmd_args(argsfile, format = "@{}", hidden = txt_args)

cmd.add(["--sources"])
cmd.add(argsfile)

# use prefer_remote = True here, it would have two following advantages
# 1. Each bucket will perform a merge on RE, which will fully utilize the high-speed network for materalizaion
Expand Down

0 comments on commit ddecda9

Please sign in to comment.