forked from TraceMachina/nativelink
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Breaking] Generalize LRE to arbitrary toolchains (TraceMachina#728)
This refactors the entire remote execution setup. We now use "base" images to supply toolchains and have wrappers to create nativelink workers from those base images. This allows us to "enrich" arbitrary toolchain containers to turn them into nativelink workers. In other words, we now have a framework to import non-Nix containers into our Nix infrastructure, such as "classic" Ubuntu-based toolchain containers. Toolchain generation is now arbitrarily fine-grained. In practice, this means that for instance the Java and C++ toolchains are now separate entities. This has a large impact on the efficiency of multi-toolchain deployments. The Kubernetes example has been updated accordingly. As a side effect of the new container structures the K8s deployment now works without root permissions in the nativelink containers. The LRE infrastructure is now treated as a special case of the new toolchain setup process. The `rbe-configs-gen` logic is now an implementation detail and the generator logic is no longer carried over into the final worker images. This brings down the image size for the LRE containers from ~2.5GB to ~1.7GB for C++ and ~600MB for Java. The slight overall reduction in container sizes is due to the omission of the Bazel executable. Bazel is required to generate the Starlark toolchain configurations but doesn't have to be present in the final worker images.
- Loading branch information
1 parent
4095e97
commit 1a43ef9
Showing
38 changed files
with
966 additions
and
237 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| --- | ||
| apiVersion: apps/v1 | ||
| kind: Deployment | ||
| metadata: | ||
| name: nativelink-worker-lre-cc | ||
| spec: | ||
| replicas: 1 | ||
| selector: | ||
| matchLabels: | ||
| app: nativelink-worker-lre-cc | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: nativelink-worker-lre-cc | ||
| spec: | ||
| containers: | ||
| - name: nativelink-worker-lre-cc | ||
| image: "localhost:5001/nativelink-worker-lre-cc:local" | ||
| env: | ||
| - name: RUST_LOG | ||
| value: warn | ||
| - name: CAS_ENDPOINT | ||
| value: nativelink-cas | ||
| - name: SCHEDULER_ENDPOINT | ||
| value: nativelink-scheduler | ||
| volumeMounts: | ||
| - name: worker-lre-cc-config | ||
| mountPath: /worker-lre-cc.json | ||
| subPath: worker-lre-cc.json | ||
| command: ["/bin/nativelink"] | ||
| args: ["/worker-lre-cc.json"] | ||
| volumes: | ||
| - name: worker-lre-cc-config | ||
| configMap: | ||
| name: worker-lre-cc |
80 changes: 80 additions & 0 deletions
80
deployment-examples/kubernetes/worker-lre-java.json.template
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| { | ||
| "stores": { | ||
| "GRPC_LOCAL_STORE": { | ||
| // Note: This file is used to test GRPC store. | ||
| "grpc": { | ||
| "instance_name": "main", | ||
| "endpoints": [ | ||
| {"address": "grpc://${CAS_ENDPOINT:-127.0.0.1}:50051"} | ||
| ], | ||
| "store_type": "cas" | ||
| } | ||
| }, | ||
| "GRPC_LOCAL_AC_STORE": { | ||
| // Note: This file is used to test GRPC store. | ||
| "grpc": { | ||
| "instance_name": "main", | ||
| "endpoints": [ | ||
| {"address": "grpc://${CAS_ENDPOINT:-127.0.0.1}:50051"} | ||
| ], | ||
| "store_type": "ac" | ||
| } | ||
| }, | ||
| "WORKER_FAST_SLOW_STORE": { | ||
| "fast_slow": { | ||
| "fast": { | ||
| "filesystem": { | ||
| "content_path": "~/.cache/nativelink/data-worker-test/content_path-cas", | ||
| "temp_path": "~/.cache/nativelink/data-worker-test/tmp_path-cas", | ||
| "eviction_policy": { | ||
| // 10gb. | ||
| "max_bytes": 10000000000, | ||
| } | ||
| } | ||
| }, | ||
| "slow": { | ||
| "ref_store": { | ||
| "name": "GRPC_LOCAL_STORE", | ||
| } | ||
| } | ||
| } | ||
| } | ||
| }, | ||
| "workers": [{ | ||
| "local": { | ||
| "worker_api_endpoint": { | ||
| "uri": "grpc://${SCHEDULER_ENDPOINT:-127.0.0.1}:50061", | ||
| }, | ||
| "cas_fast_slow_store": "WORKER_FAST_SLOW_STORE", | ||
| "upload_action_result": { | ||
| "ac_store": "GRPC_LOCAL_AC_STORE", | ||
| }, | ||
| "work_directory": "~/.cache/nativelink/work", | ||
| "platform_properties": { | ||
| "cpu_count": { | ||
| "query_cmd": "nproc" | ||
| }, | ||
| "OSFamily": { | ||
| "values": ["Linux"] | ||
| }, | ||
| "container-image": { | ||
| "values": [ | ||
| // WARNING: This is *not* the container that is actually deployed | ||
| // here. The generator container in this example was | ||
| // `rbe-autogen-lre-java:<sometag>` and the platform was modified | ||
| // after the fact to be `lre-java:<sometag>`. The deployed container | ||
| // we use as worker is `nativelink-worker-lre-java:<sometag>` which | ||
| // is a completely separate extension of the `lre-java` base image. | ||
| // | ||
| // Treat the `docker//:...` string below as nothing more than a raw | ||
| // string that is matched by the scheduler against the value | ||
| // specified in the `exec_properties` of the corresponding platform | ||
| // at `local-remote-execution/generated-java/config/BUILD`. | ||
| "docker://lre-java:__LRE_JAVA_TOOLCHAIN_TAG__", | ||
| ] | ||
| } | ||
| } | ||
| } | ||
| }], | ||
| "servers": [] | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| --- | ||
| apiVersion: apps/v1 | ||
| kind: Deployment | ||
| metadata: | ||
| name: nativelink-worker-lre-java | ||
| spec: | ||
| replicas: 1 | ||
| selector: | ||
| matchLabels: | ||
| app: nativelink-worker-lre-java | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: nativelink-worker-lre-java | ||
| spec: | ||
| containers: | ||
| - name: nativelink-worker-lre-java | ||
| image: "localhost:5001/nativelink-worker-lre-java:local" | ||
| env: | ||
| - name: RUST_LOG | ||
| value: warn | ||
| - name: CAS_ENDPOINT | ||
| value: nativelink-cas | ||
| - name: SCHEDULER_ENDPOINT | ||
| value: nativelink-scheduler | ||
| volumeMounts: | ||
| - name: worker-lre-java-config | ||
| mountPath: /worker-lre-java.json | ||
| subPath: worker-lre-java.json | ||
| command: ["/bin/nativelink"] | ||
| args: ["/worker-lre-java.json"] | ||
| volumes: | ||
| - name: worker-lre-java-config | ||
| configMap: | ||
| name: worker-lre-java |
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.