Tracking issue for impact of 2024 MacOS fleet expansion #1981

fweikert · 2024-07-05T14:10:34Z

We've moved most of our MacOS CI workload from lab machines to a larger number of less powerful VMs (#1708).

Please reply to this issue if you've encountered any MacOS issues related to the migration.

meteorcloudy · 2024-07-08T13:03:01Z

FYI @keith , not sure the latest rules_apple failures are related to this or not: https://buildkite.com/bazel/rules-apple-darwin/builds/9311

keith · 2024-07-08T18:20:07Z

yea they are, looks like there was an Xcode update as part of this, which is good, tracking fix here bazelbuild/rules_apple#2488

Wyverald · 2024-07-08T18:33:28Z

there are some BEP-related errors in this presubmit that only happen on macOS and don't go away after retries: https://buildkite.com/bazel/bcr-presubmit/builds/6541#019092db-1572-49fe-b7e3-2f489764d67a

ahumesky · 2024-07-09T00:07:11Z

It looks like rules_jvm_external is having trouble on the new macs with whatever version of java they have:
https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/3973#01909143-b698-405c-b925-02461d6dbdab
https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/3973#01909085-2f47-4ce2-a9f9-21637ede87eb

(09:35:35) ERROR: An error occurred during the fetch of repository 'rules_android_maven':
   Traceback (most recent call last):
	File "/private/var/tmp/_bazel_buildkite/e429569ac118389554140bb6d293e1ed/external/rules_jvm_external/coursier.bzl", line 985, column 38, in _coursier_fetch_impl
		dep_tree = make_coursier_dep_tree(
	File "/private/var/tmp/_bazel_buildkite/e429569ac118389554140bb6d293e1ed/external/rules_jvm_external/coursier.bzl", line 889, column 30, in make_coursier_dep_tree
		if parse_java_version(exec_result.stdout + exec_result.stderr) > 8:
	File "/private/var/tmp/_bazel_buildkite/e429569ac118389554140bb6d293e1ed/external/rules_jvm_external/private/java_utilities.bzl", line 48, column 29, in parse_java_version
		return get_major_version(first_line[i:j])
	File "/private/var/tmp/_bazel_buildkite/e429569ac118389554140bb6d293e1ed/external/rules_jvm_external/private/java_utilities.bzl", line 32, column 19, in get_major_version
		return int(java_version.split(".")[0])
Error in int: empty string

I can try updating the version of rules_jvm_external, it looks like the latest version doesn't have this parse_java_version function

ahumesky · 2024-07-09T00:13:32Z

I also notice that the mac arm64 tests on rules_android are failing with "Bad CPU type in executable" when it tries to execute aapt2, I haven't looked into that further
https://buildkite.com/bazel/rules-android/builds/2789#01908fb0-ef50-4143-b77d-c5e335830cc4

Wyverald · 2024-07-09T00:50:37Z

hmm. Also seeing macOS jobs on Bazel 6.x stuck on Waiting for remote cache: 1 upload after a successful build (see https://buildkite.com/bazel/bcr-presubmit/builds/6573#019094af-bd7d-44a6-98f5-a9068585e48e). Bazel 7.x are okay, so this might have been a bug in Bazel 6.x that got fixed.

ahumesky · 2024-07-09T02:36:05Z

bazelbuild/rules_android#244 fixes the parse_java_version failure

ahumesky · 2024-07-09T03:21:10Z

I also notice that the mac arm64 tests on rules_android are failing with "Bad CPU type in executable" when it tries to execute aapt2, I haven't looked into that further https://buildkite.com/bazel/rules-android/builds/2789#01908fb0-ef50-4143-b77d-c5e335830cc4

Ok so downloading the android sdk build tools for mac:
~/android-sdk$ REPO_OS_OVERRIDE=macosx ./cmdline-tools/7.0/bin/sdkmanager --sdk_root=$HOME/android-sdk-macosx --install "build-tools;30.0.3" "build-tools;35.0.0"

and then examining the appt2 binary for the currently installed build tools vesrion 30.0.3:
~/android-sdk$ file ~/android-sdk-macosx/build-tools/30.0.3/aapt2 /usr/local/google/home/ahumesky/android-sdk-macosx/build-tools/30.0.3/aapt2: Mach-O 64-bit x86_64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|WEAK_DEFINES|BINDS_TO_WEAK|PIE|HAS_TLV_DESCRIPTORS>

The currently installed 30.0.3 has only x86_64, and the latest version is both a x86_64 and arm64 binary.

The arm64 tests were passing before, but I'm not sure how or why. I guess there was some x86_64 emulation going on that's not enabled on the new runners? In any case, we should just update the android sdk build tools on the CI images (https://github.com/bazelbuild/continuous-integration/blob/master/macos/mac-android.sh etc) to the latest build tools 35.0.0

meteorcloudy · 2024-07-09T12:24:00Z

there are some BEP-related errors in this presubmit that only happen on macOS and don't go away after retries: https://buildkite.com/bazel/bcr-presubmit/builds/6541#019092db-1572-49fe-b7e3-2f489764d67a

Probably unrelated to macOS migration, see bazelbuild/bazel-central-registry#2373 (comment)

meteorcloudy · 2024-07-09T12:44:02Z

If your builds on Intel macOS frequently run into timeouts like bazel-contrib/rules_go#3969 (comment) after this change, this is because the machines in the new fleet is less powerful than previous ones (2 core Mac Mini vs 20 core iMac Pro)

Consider applying the following changes:

Migrate to the Apple Silicon platform (macos_arm64), Intel macOS is the legacy platform anyway.
~~Add --local_test_jobs=2 to test flags to reduce the number of tests running in parallel~~ This is already done automatically by CI.
Shard your task by adding shards:<N> in your presubmit.yml file (Please don't set N too large since we still have limited amount of machines available)
Bump your test size to allow longer timeout

meteorcloudy · 2024-07-09T13:34:14Z

hmm. Also seeing macOS jobs on Bazel 6.x stuck on Waiting for remote cache: 1 upload after a successful build (see https://buildkite.com/bazel/bcr-presubmit/builds/6573#019094af-bd7d-44a6-98f5-a9068585e48e).

This is caused by --experimental_remote_cache_async, and there is still a bug in bazel 6.x. We can remove this flag from default CI step. @fweikert

jsharpe · 2024-07-09T16:11:00Z

rules_foreign_cc is having issues with the new macOS runners too: https://buildkite.com/bazel/rules-foreign-cc/builds/5745#_

meteorcloudy · 2024-07-09T16:33:05Z

rules_foreign_cc is having issues with the new macOS runners too: https://buildkite.com/bazel/rules-foreign-cc/builds/5745#_

Probably mesonbuild/meson#12282 ?

jsharpe · 2024-07-09T16:43:33Z

rules_foreign_cc is having issues with the new macOS runners too: buildkite.com/bazel/rules-foreign-cc/builds/5745_

Probably mesonbuild/meson#12282 ?

Looks like it but the fix is probably something that the apple toolchain should apply, as rules_foreign_cc can't really know the version of xcode and when its needed? Any opinions @keith?

…sue on new mac CI machines. See bazelbuild/continuous-integration#1981 (comment) Closes #244 Closes #242 PiperOrigin-RevId: 650660707 Change-Id: I3e93a893684786951e1f345ee9749ee62ec7049e

keith · 2024-07-09T17:32:51Z

hrm seems like the repo's files should deal with this if they need it? theoretically you could know what the current xcode version was if you wanted, but unless you're setting that version wrong today I don't think that would help with downstream issues?

- Upgraded Bazel to 7.2.1 to include remote cache fixes - Backport changes to adapt bazelbuild/continuous-integration#1981 --------- Co-authored-by: Googler <[email protected]>

brentleyjones · 2024-07-10T17:06:56Z

WARNING: ignoring JAVA_TOOL_OPTIONS in environment.
--
  | Jul 10 18:56:13.449 ERROR bazelci_agent::artifact::upload: Failed to read /tmp/tmpft73_y33/test_bep.json
  |  
  | Caused by:
  | No such file or directory (os error 2)
  | Jul 10 18:56:13.451 ERROR bazelci_agent::artifact::upload: Failed to read /tmp/tmpft73_y33/test_bep.json
  |  
  | Caused by:
  | No such file or directory (os error 2)
  | Jul 10 18:56:13.451 ERROR bazelci_agent::artifact::upload: Failed to read /tmp/tmpft73_y33/test_bep.json
  |  
  | Caused by:
  | No such file or directory (os error 2)
  | Jul 10 18:56:13.451 ERROR bazelci_agent::artifact::upload: Failed to read /tmp/tmpft73_y33/test_bep.json
  |  
  | Caused by:
  | No such file or directory (os error 2)
  | Jul 10 18:56:13.451 ERROR bazelci_agent::artifact::upload: Failed to read /tmp/tmpft73_y33/test_bep.json
  |  
  | Caused by:
  | No such file or directory (os error 2)
  | Error: Failed to read /tmp/tmpft73_y33/test_bep.json
  |  
  | Caused by:
  | No such file or directory (os error 2)
  | (18:56:15) WARNING: Option 'experimental_remote_build_event_upload' is deprecated: Use --remote_build_event_upload instead

https://buildkite.com/bazel/bcr-presubmit/builds/6641#01909d90-e33f-49cf-ba58-fcd588073673/415-445

lkassar-stripe · 2024-07-11T00:12:23Z

Any chance I could get some help with one of the MacOS failures on my PR? It seems like its probably related to this issue, please let me know if there's any configurations I should be changing here. This is my first contribution here so any advice welcome! bazel-contrib/bazel-gazelle#1822

keith · 2024-07-11T00:15:20Z

Likely unrelated, I don't have access to retry it but hopefully someone can

meteorcloudy · 2024-07-11T08:56:05Z

@lkassar-stripe The timeout is likely caused by the Bazel binary cannot reach to the network in an ipv6-only network.
See my comment here: bazel-contrib/rules_go#3969 (comment)

Unfortunately, the test framework needs to inject --host_jvm_args=-Djava.net.preferIPv6Addresses=true as a startup option somewhere. If this is urgent, you can temporarily switch back to macos_legacy or macos_arm64_legacy platform to use the old Mac fleet. But we will remove them soon in the future.

meteorcloudy · 2024-07-11T08:57:08Z

ERROR bazelci_agent::artifact::upload: Failed to read /tmp/tmpft73_y33/test_bep.json

@coeuvre Do you know what's happening?

I'm also seeing

Command '['/tmp/tmpft73_y33/bazelci-agent', 'artifact', 'upload', '--debug', '--delay=5', '--mode=buildkite', '--build_event_json_file=/tmp/tmpft73_y33/test_bep.json']' returned non-zero exit status 1.

from time to time.

coeuvre · 2024-07-11T09:29:12Z

Hard to tell from this error message. Do you have link to the build which produced the error?

meteorcloudy · 2024-07-11T11:09:07Z

Yes, this one #1981 (comment)

sluongng · 2024-07-18T16:23:23Z

@meteorcloudy I am having problem in Bazel.git as well.

Currently on my MacOS, to run a shell test, I would have to apply this patch

diff --git a/src/test/shell/bazel/remote_helpers.sh b/src/test/shell/bazel/remote_helpers.sh
index c6eba4dcad..5cc3b3036a 100755
--- a/src/test/shell/bazel/remote_helpers.sh
+++ b/src/test/shell/bazel/remote_helpers.sh
@@ -16,7 +16,7 @@

 set -eu

-setup_localjdk_javabase
+# setup_localjdk_javabase

 # Serves $1 as a file on localhost:$nc_port.  Sets the following variables:
 #   * nc_port - the port nc is listening on.
diff --git a/src/test/shell/testenv.sh.tmpl b/src/test/shell/testenv.sh.tmpl
index 4058bcf74d..efa11a368d 100755
--- a/src/test/shell/testenv.sh.tmpl
+++ b/src/test/shell/testenv.sh.tmpl
@@ -390,11 +390,11 @@ EOF
     echo "startup --install_base=$TEST_INSTALL_BASE" >> $TEST_TMPDIR/bazelrc
   fi

-  if is_darwin; then
-    echo "Add flags to prefer ipv6 network"
-    echo "startup --host_jvm_args=-Djava.net.preferIPv6Addresses=true" >> $TEST_TMPDIR/bazelrc
-    echo "build --jvmopt=-Djava.net.preferIPv6Addresses" >> $TEST_TMPDIR/bazelrc
-  fi
+  # if is_darwin; then
+  #   echo "Add flags to prefer ipv6 network"
+  #   echo "startup --host_jvm_args=-Djava.net.preferIPv6Addresses=true" >> $TEST_TMPDIR/bazelrc
+  #   echo "build --jvmopt=-Djava.net.preferIPv6Addresses" >> $TEST_TMPDIR/bazelrc
+  # fi
 }

This is the test command I am running on my mac bazel test src/test/shell/bazel/remote:remote_execution_test

Without commenting out the ipv6 flags, I would get

-- Test log: -----------------------------------------------------------
$TEST_TMPDIR defined: output root default is '/private/var/tmp/_bazel_sluongng/4528d0377c5d584786af4e2410173691/sandbox/darwin-sandbox/4912/execroot/_main/_tmp/ee9047f7e54192d1c96723be06c02001' and max_idle_secs default is '15'.
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 71d0a02e-0619-44f8-abb8-383473c5d0ff
Computing main repo mapping:
Computing main repo mapping:
ERROR: Error computing the main repository mapping: Error accessing registry https://bcr.bazel.build/: Failed to fetch registry file https://bcr.bazel.build/modules/rules_proto/4.0.0/MODULE.bazel: No route to host
------------------------------------------------------------------------

meteorcloudy · 2024-07-19T08:44:46Z

Yeah, this is the same problem. We probably should also remove --nosystem_rc from https://cs.opensource.google/bazel/bazel/+/master:src/test/shell/bin/bazel;l=26 and let the Bazel binary inside the test to pick up the ipv6 flag from /private/etc/bazel.bazelrc on the Mac VM, so that user builds are not affected.

/cc @fweikert

sluongng · 2024-07-19T09:58:39Z

I wonder if it's a good opportunity to make this a platform constraint somehow and augment the tests with a select.

fweikert · 2024-07-19T17:38:05Z

Filed #2003 to track the Intel MacOS problem with test analytics (caused by this migration).

dws · 2024-07-23T00:29:06Z

Possible MacOS issue here: https://buildkite.com/bazel/bazel-bazel-github-presubmit/builds/22496#0190dcaf-a704-424e-9441-dd7426239485
CI passed on a rerun.

ERROR: /Users/buildkite/builds/bk-macos-pln3-1zzn/bazel/bazel-bazel-github-presubmit/src/main/java/com/google/devtools/build/lib/bazel/BUILD:219:12: Building deploy jar src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar [for tool] failed: I/O exception during sandboxed execution: 19 errors during bulk transfer:

sluongng · 2024-07-23T13:01:32Z

Im also getting a new persistent issue on a specific MacOS shard:

https://buildkite.com/bazel/bazel-bazel-github-presubmit/builds/22503#0190dede-b514-430d-9794-2bb5847c81fe/383-417

Error in download_and_extract: java.io.IOException: Error extracting /private/var/tmp/_bazel_buildkite/dc75035d57eb8cda9b01612d7ce9b862/external/rules_java~~toolchains~remotejdk17_macos/temp5719727540427988246/zulu17.50.19-ca-jdk17.0.11-macosx_x64.tar.gz to /private/var/tmp/_bazel_buildkite/dc75035d57eb8cda9b01612d7ce9b862/external/rules_java~~toolchains~remotejdk17_macos/temp5719727540427988246: Corrupted TAR archive.

I did try to push another commit to re-trigger CI for this PR but the same issue happened again.

meteorcloudy · 2024-07-23T13:05:07Z

And it happened on the same machine bk-macos-pln3-90qy (build 1, build 2). @fweikert Can we report such errors back to the internal team offering the VMs?

FaBrand · 2024-07-23T14:43:01Z

For my builds a few of the macOS ones failed with Exited with status -1 (agent lost)
https://buildkite.com/bazel/bazel-bazel-github-presubmit/builds/22506#0190dfc4-1964-461d-945a-8c3002019307

Some even in the workspace preparation Stage

Or in Waiting for agent

meteorcloudy · 2024-07-23T14:46:57Z

@FaBrand We are deploying an update to the VMs, sorry for the inconvenience.

FaBrand · 2024-07-23T14:49:04Z

@FaBrand We are deploying an update to the VMs, sorry for the inconvenience.

Thanks for the Info @meteorcloudy :) Do i have to do something, to retrigger my checks?

meteorcloudy · 2024-07-23T15:36:29Z

There is already a retry job scheduled for each lost agent, they will run as soon as the VMs are online.

criemen · 2024-07-23T23:01:03Z

I'm not sure if it's related to the new Mac CI workers, but as this issue is prominently featured on buildkite, I'll report it here. If you think it's unrelated, I'm happy to report this as seperate issue elsewhere.

I'm hitting

(00:54:28) ERROR: /Users/buildkite/builds/bk-macos-pln3-yigd/bazel/bazel-bazel-github-presubmit/src/test/java/com/google/devtools/build/lib/skyframe/BUILD:1527:10: Building src/test/java/com/google/devtools/build/lib/skyframe/LocalRepositoryLookupFunctionTest.jar (1 source file) failed: Worker process returned an unparseable WorkResponse!
Did you try to print something to stdout? Workers aren't allowed to do this, as it breaks the protocol between Bazel and the worker process.
---8<---8<--- Start of response ---8<---8<---
Not UTF-8, printing as hex
02 18 01 02 18 03 02 18  02 02 18 04 02 18 06 02  |........ ........|
18 05 02 18 07 02 18 01  02 18 02 02 18 03 02 18  |........ ........|
04 02 18 06 02 18 05 02  18 07 02 18 08 02 18 09  |........ ........|
02 18 0A 02 18 0C 02 18  0B 02 18 0D 02 18 0E 02  |........ ........|
18 0F 02 18 11 02 18 10  02 18 12 02 18 13 02 18  |........ ........|
14 02 18 15 23 0A 23 20  41 20 66 61 74 61 6C 20  |....#.#  A fatal |
65 72 72 6F 72 20 68 61  73 20 62 65 65 6E 20 64  |error ha s been d|
65 74 65 63 74 65 64 20                           |etected          |
---8<---8<--- End of response ---8<---8<---
---8<---8<--- Exception details ---8<---8<---
java.io.IOException: Worker process for Javac has died
...

on https://buildkite.com/bazel/bazel-bazel-github-presubmit/builds/22520#0190e1c7-b7fe-4b6b-934c-624bee706f7d which is an exception I've never come across yet.

FaBrand · 2024-07-24T07:59:42Z

@meteorcloudy FYI There is one shard that wasn't retried again after 4 Agent losses:
https://buildkite.com/bazel/bazel-bazel-github-presubmit/builds/22506#0190e052-2f69-493f-bb0a-c11b6f93dcf1

FaBrand · 2024-07-24T09:55:58Z

@meteorcloudy FYI There is one shard that wasn't retried again after 4 Agent losses: https://buildkite.com/bazel/bazel-bazel-github-presubmit/builds/22506#0190e052-2f69-493f-bb0a-c11b6f93dcf1

I guess someone retriggered it, thank you dear anonymous helper 💚

Related #1981

albertocavalcante · 2024-07-29T21:19:40Z

bazel/buildtools: `macOS arm64`

I'm facing an error on this build:

/bin/bash: external/jq_darwin_arm64/jq: Bad CPU type in executable

More info:

ERROR: /Users/buildkite/builds/bk-macos-arm64-cn9q/bazel/buildtools/buildifier/npm/BUILD.bazel:43:21: Jq buildifier/npm/package.json failed: (Exit 126): bash failed: error executing Jq command (from target //buildifier/npm:package)
--
  | (cd /private/var/tmp/_bazel_buildkite/4e34735c8e9a8b6c609838cd5f90f526/sandbox/darwin-sandbox/629/execroot/buildtools && \
  | exec env - \
  | /bin/bash -c 'external/jq_darwin_arm64/jq  '\''$ARGS.named.STAMP as $stamp\|.version = ($stamp.BUILD_SCM_VERSION // "0.0.0")'\'' '\''buildifier/npm/package.json'\'' > bazel-out/darwin_arm64-fastbuild/bin/buildifier/npm/package.json')
  | # Configuration: 05e9e48b199e39946b162f1e18d6c3760aa2e4131e967a19ba1e1be8890508b2
  | # Execution platform: @@internal_platforms_do_not_use//host:host
  |  
  | Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
  | /bin/bash: external/jq_darwin_arm64/jq: Bad CPU type in executable

meteorcloudy · 2024-07-30T11:42:12Z

We need the fix from bazelbuild/bazel#18444, which is only included in 7.3.0. We can pin Bazel version for buildtools to 7.3.0rc1. /cc @vladmos

pat-jpnk · 2024-08-08T04:44:29Z

Facing the following build error for MacOS, related to this merge request

meteorcloudy · 2024-09-10T15:21:47Z

CI infra problems for the new macOS fleet should have been addressed already, closing this one for now

This marks the end of bazelbuild#1981

This marks the end of #1981

ahumesky mentioned this issue Jul 9, 2024

Update Android build tools to latest version 35.0.0 #1991

Closed

This was referenced Jul 9, 2024

Add a full fs.FS implementation to runfiles bazel-contrib/rules_go#3969

Merged

[email protected] bazelbuild/bazel-central-registry#2373

Closed

This was referenced Jul 9, 2024

[7.3.0] Fix CI for release-7.3.0 bazelbuild/bazel#22976

Merged

Flaky Mac test log upload #1963

Closed

sluongng mentioned this issue Jul 9, 2024

Dependencies upgrades bazel-contrib/bazel-gazelle#1838

Closed

FaBrand mentioned this issue Jul 23, 2024

Fix inconsistent virtual include paths bazelbuild/bazel#23071

Closed

meteorcloudy mentioned this issue Jul 25, 2024

Remove unused macOS platforms #2011

Merged

meteorcloudy added a commit that referenced this issue Jul 25, 2024

Remove unused macOS platforms (#2011)

466722f

Related #1981

albertocavalcante mentioned this issue Jul 29, 2024

Replace usage deprecated io/iotil bazelbuild/buildtools#1279

Merged

jsharpe mentioned this issue Aug 1, 2024

[Bazel CI] CcMesonMakeRule Fails with Linker Error, Build Fails bazel-contrib/rules_foreign_cc#1244

Closed

hvadehra mentioned this issue Aug 8, 2024

Update java_tools v13.7 / rules_java 7.9.0 bazelbuild/bazel#23237

Closed

meteorcloudy closed this as completed Sep 10, 2024

fweikert added a commit to fweikert/continuous-integration that referenced this issue Sep 10, 2024

Remove code related to obsolete lab MacOS machines

d9a1ab0

This marks the end of bazelbuild#1981

fweikert added a commit to fweikert/continuous-integration that referenced this issue Sep 10, 2024

Remove code related to obsolete lab MacOS machines

4640f4a

This marks the end of bazelbuild#1981

fweikert mentioned this issue Sep 10, 2024

Remove code related to obsolete MacOS lab machines #2036

Merged

fweikert added a commit to fweikert/continuous-integration that referenced this issue Sep 10, 2024

Remove code related to obsolete MacOS lab machines

45513bd

This marks the end of bazelbuild#1981

fweikert added a commit that referenced this issue Sep 10, 2024

Remove code related to obsolete MacOS lab machines (#2036)

6a8dad1

This marks the end of #1981

Tracking issue for impact of 2024 MacOS fleet expansion #1981

Tracking issue for impact of 2024 MacOS fleet expansion #1981

Comments

fweikert commented Jul 5, 2024

meteorcloudy commented Jul 8, 2024

keith commented Jul 8, 2024

Wyverald commented Jul 8, 2024

ahumesky commented Jul 9, 2024

ahumesky commented Jul 9, 2024 • edited Loading

Wyverald commented Jul 9, 2024

ahumesky commented Jul 9, 2024

ahumesky commented Jul 9, 2024 • edited Loading

meteorcloudy commented Jul 9, 2024

meteorcloudy commented Jul 9, 2024 • edited Loading

meteorcloudy commented Jul 9, 2024

jsharpe commented Jul 9, 2024

meteorcloudy commented Jul 9, 2024

jsharpe commented Jul 9, 2024

keith commented Jul 9, 2024

brentleyjones commented Jul 10, 2024

lkassar-stripe commented Jul 11, 2024

keith commented Jul 11, 2024

meteorcloudy commented Jul 11, 2024

meteorcloudy commented Jul 11, 2024

coeuvre commented Jul 11, 2024

meteorcloudy commented Jul 11, 2024

sluongng commented Jul 18, 2024

meteorcloudy commented Jul 19, 2024

sluongng commented Jul 19, 2024

fweikert commented Jul 19, 2024

dws commented Jul 23, 2024

sluongng commented Jul 23, 2024 • edited Loading

meteorcloudy commented Jul 23, 2024

FaBrand commented Jul 23, 2024 • edited Loading

meteorcloudy commented Jul 23, 2024

FaBrand commented Jul 23, 2024 • edited Loading

meteorcloudy commented Jul 23, 2024

criemen commented Jul 23, 2024 • edited Loading

FaBrand commented Jul 24, 2024

FaBrand commented Jul 24, 2024 • edited Loading

albertocavalcante commented Jul 29, 2024

bazel/buildtools: macOS arm64

meteorcloudy commented Jul 30, 2024

pat-jpnk commented Aug 8, 2024

meteorcloudy commented Sep 10, 2024

ahumesky commented Jul 9, 2024 •

edited

Loading

ahumesky commented Jul 9, 2024 •

edited

Loading

meteorcloudy commented Jul 9, 2024 •

edited

Loading

sluongng commented Jul 23, 2024 •

edited

Loading

FaBrand commented Jul 23, 2024 •

edited

Loading

FaBrand commented Jul 23, 2024 •

edited

Loading

criemen commented Jul 23, 2024 •

edited

Loading

FaBrand commented Jul 24, 2024 •

edited

Loading

bazel/buildtools: `macOS arm64`