ARROW-10792: [Rust] [CI] Modularize builds for faster build and small…

…er caches This PR refactors the CI with the purpose of making it faster, easier to extend, and more robust. The main consequences of this PR are: 1. build time of non-integration jobs reduced by half (10m vs 20m) 2. coverage is now part of all builds 3. integration with the c data interface is now part of all builds 4. significantly reduced the risk of cache misses, and the size of the caches 5. reduced complexity by removing the need to build a docker image 6. macos now have caching in place, like other builds 7. significantly easier to extend to other architectures, os, etc. The overall design of this PR is that most steps of the CI process should be independent of all others as much as possible, so that * they can run in parallel * it is clearer where the failure is * it is easier to extend the build to other variations (arch, os, flags, etc.) Intrinsically, the CI pipeline is just a DAG where certain jobs benefit from shared compute and thus depend on common nodes, while others run independently. This PR re-writes most of our CI as simple github jobs that share caches whenever makes sense, thereby describing the aforementioned DAG. Consequently, many of the sequential steps are now executed in parallel, causing the items describe above. There are two tasks depending on rust that are not in the `rust.yaml`: * lint, that happens via archery * integration tests, that happens via archery The bottleneck of a green pipeline are the integration tests, that currently re-compile a significant number of components (as I already flagged on the mailing list), but at least we get faster failures for compilation (2.5m) and clippy (4.5m). Closes apache#8821 from jorgecarleitao/build_tests Authored-by: Jorge C. Leitao <[email protected]> Signed-off-by: Andrew Lamb <[email protected]>
sirensolutions · Jun 7, 2021 · 8ca4e1e · 8ca4e1e
1 parent f0b2b60
commit 8ca4e1e
Show file tree

Hide file tree

Showing 9 changed files with 318 additions and 479 deletions.
diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
diff --git a/.github/workflows/rust_cron.yml b/.github/workflows/rust_cron.yml
diff --git a/ci/docker/debian-10-rust.dockerfile b/ci/docker/debian-10-rust.dockerfile
diff --git a/ci/scripts/rust_coverage.sh b/ci/scripts/rust_coverage.sh
diff --git a/ci/scripts/rust_lint.sh b/ci/scripts/rust_lint.sh
diff --git a/ci/scripts/rust_pyarrow_integration.sh b/ci/scripts/rust_pyarrow_integration.sh
diff --git a/ci/scripts/rust_test.sh b/ci/scripts/rust_test.sh
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -102,7 +102,6 @@ x-hierarchy:
   - debian-java:
     - debian-java-jni
   - debian-js
-  - debian-rust
   - fedora-cpp:
     - fedora-python
   - ubuntu-cpp:
@@ -903,35 +902,6 @@ services:
       /bin/bash -c "
         /arrow/ci/scripts/r_sanitize.sh /arrow"
 
-  ################################ Rust #######################################
-
-  debian-rust:
-    # Usage:
-    #   docker-compose build debian-rust
-    #   docker-compose run debian-rust
-    image: ${REPO}:${ARCH}-debian-10-rust-${RUST}
-    build:
-      context: .
-      dockerfile: ci/docker/debian-10-rust.dockerfile
-      cache_from:
-        - ${REPO}:${ARCH}-debian-10-rust-${RUST}
-      args:
-        arch: ${ARCH}
-        rust: ${RUST}
-    shm_size: *shm-size
-    environment:
-      CARGO_HOME: /rust/cargo
-      CARGO_TARGET_DIR: /build/rust
-    volumes: &rust-volumes
-      - .:/arrow:delegated
-      - ${DOCKER_VOLUME_DIRECTORY:-.docker}/${ARCH}-debian-${DEBIAN}-rust:/build:delegated
-    command: &rust-command >
-      /bin/bash -c "
-        echo ${RUST} > /arrow/rust/rust-toolchain &&
-        if [ ! -d "/build/rust" ]; then mv /rust/target /build/rust; fi &&
-        /arrow/ci/scripts/rust_build.sh /arrow /build &&
-        /arrow/ci/scripts/rust_test.sh /arrow /build"
-
   ################################# Go ########################################
 
   debian-go: