perf: reduce memory usage of bes-uploader

[christianscott]

Description of the bug:

we've been able to improve the overall performance of our builds by improving the throughput of bes-uploader. when events are not processed quickly, the eventQueue and ackQueue may grow faster than they're cleared, meaning these queues consume more and more of the heap. sometimes this causes bazel to OOM, but in other cases the build is slowed down because of extra competition for memory.

we've seen two reasons for bes-uploader processing events slowly:

1. slow remote server. we've seen this when the CAS is slow, but I imagine a slow BES would have the same effect
2. bes-uploader does too much work, see DigestUtils: avoid throwing on invalid digest function name #20574 and ByteStreamBuildEventArtifactUploader: skip reading metadata for files that won't be uploaded #20575

there are things we can do to address both of these, but it would be nice if the bes-uploader wasn't able to cause the rest of the build to perform poorly, even if it can't clear the events quickly.

note that bes_upload_mode=fully-async does not help because the events still need to be stored in memory.

some ideas:

1. reduce size of SendRegularBuildCommand by using PathConverter ASAP. if paths were converted when an event is pushed to the queue, then PathConverter instances could be collected immediately. some PathConverter instances for our monorepo are 22mb (see screenshot below)
2. serialize events before appending to the eventQueue
3. offload some of the queue to disk (maybe using mmap)

Which category does this issue belong to?

Performance

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

macos, linux

What is the output of bazel info release?

6.4.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[christianscott]

I wonder, how crucial is it for all build events to be submitted to the remote server? Should the build crash if events can't be submitted to the remote server? How bad is it to drop events if there's a risk of crashing the build? I wonder bazel could stop accepting events if the queue grew beyond some length.

[tjgq]

We should fix #20576 first and then reanalyze this. My hope is that once we're no longer digesting files unnecessarily in the BES uploader, both the queuing and memory problems go away.

[joeljeske]

I am seeing intermittent OOMs due to growing size of BES uploader queue in 7.1.1

Is there any progress on this issue, or any information I can provide to assist?