ARROW-12204: [Rust][CI] Reduce size of Rust build artifacts in integration test #9889
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alamb
commented
Apr 5, 2021
| @@ -21,6 +21,10 @@ set -ex | |||
|
|
|||
| source_dir=${1}/rust | |||
|
|
|||
| # This file is used to build the rust binaries needed for the | |||
There was a problem hiding this comment.
Doing a full rust build as part of the integration test appears to be unecessary
| @@ -15,7 +15,7 @@ | |||
| // specific language governing permissions and limitations | |||
| // under the License. | |||
|
|
|||
| //! Implementation of DataFrame API | |||
| //! Implementation of DataFrame API. | |||
There was a problem hiding this comment.
this is here to force the rust + integration tests to run
|
By the way, you can run integration tests locally using |
|
Se here for installing Archery. |
alamb
force-pushed
the
alamb/reduce_integration_size
branch
from
40e45b1 to
6317315
Compare
jorgecarleitao
approved these changes
Apr 6, 2021
| strategy: | ||
| matrix: | ||
| arch: [amd64] | ||
| rust: [nightly-2020-11-24] |
There was a problem hiding this comment.
Should we change this to stable(or have both), since this is the target version?
There was a problem hiding this comment.
yes, that is a good idea. I will make the change
pachadotdev
pushed a commit
to pachadotdev/arrow
that referenced
this pull request
Apr 6, 2021
…ation test # Rationale The [integration test](https://github.com/apache/arrow/pull/9884/checks?check_run_id=2263730460) has a fixed size builder docker image and has builds from several Arrow implementations. The Rust build artifacts (compiled binaries) in the integration tests still consume ~ 1GB of space even after apache#9879 (see @pitrou 's comment on apache#9884 (comment)). # Changes 1. Only build the `arrow-integration-testing` crate (which has the binaries needed for integration testing) rather than all of them 2. Remove incremental build artifacts 3. Moves `cargo check` to ensure we can build arrow without default features to a github action and out of the integration test This should both speed up the rust build as part of integration testing as well as reduce the amount of disk space required in the builder (both peak and after the rust build). # Results According to this [log](https://pipelines.actions.githubusercontent.com/CKhEcbzaHKPSiCoVWyWCFardoFMA9UN2zDnXYiBlRYO31o0IIv/_apis/pipelines/1/runs/179726/signedlogcontent/3?urlExpires=2021-04-05T12%3A37%3A13.1888704Z&urlSigningMethod=HMACV1&urlSignature=zROZZZham0BGH5f2vbe65n3DL13xXLgD9RJXybRqBAA%3D) after this PR's changes, the rust artifacts are much smaller. Note we could still save space by removing the rust toolchain and `.cargo` files but I think that can be an optimization for another time: ``` 2021-04-05T11:30:22.6164120Z 244M /opt/go/pkg 2021-04-05T11:30:22.6164599Z 393M /opt/go 2021-04-05T11:30:22.7201596Z 110M /opt/conda/envs/arrow/share 2021-04-05T11:30:22.7220002Z 171M /opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu/9.3.0 2021-04-05T11:30:22.7220911Z 171M /opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu 2021-04-05T11:30:22.7221489Z 171M /opt/conda/envs/arrow/libexec/gcc 2021-04-05T11:30:22.7221945Z 186M /opt/conda/envs/arrow/libexec 2021-04-05T11:30:22.7314210Z 186M /opt/conda/envs/arrow/lib/valgrind 2021-04-05T11:30:22.8109246Z 129M /opt/conda/envs/arrow/lib/python3.8/site-packages 2021-04-05T11:30:22.8290900Z 177M /opt/conda/envs/arrow/lib/python3.8 2021-04-05T11:30:22.8636030Z 1.8G /opt/conda/envs/arrow/lib 2021-04-05T11:30:22.9861104Z 171M /opt/conda/envs/arrow/include/boost 2021-04-05T11:30:23.0128341Z 260M /opt/conda/envs/arrow/include 2021-04-05T11:30:23.0163805Z 417M /opt/conda/envs/arrow/bin 2021-04-05T11:30:23.0449297Z 102M /opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr/lib64 2021-04-05T11:30:23.0525177Z 133M /opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr 2021-04-05T11:30:23.0526023Z 139M /opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot 2021-04-05T11:30:23.0527418Z 189M /opt/conda/envs/arrow/x86_64-conda-linux-gnu 2021-04-05T11:30:23.0612702Z 108M /opt/conda/envs/arrow/jre/lib 2021-04-05T11:30:23.0613140Z 109M /opt/conda/envs/arrow/jre 2021-04-05T11:30:23.0618402Z 3.2G /opt/conda/envs/arrow 2021-04-05T11:30:23.0618775Z 3.2G /opt/conda/envs 2021-04-05T11:30:23.1055113Z 159M /opt/conda/lib 2021-04-05T11:30:23.5619580Z 146M /opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib/valgrind 2021-04-05T11:30:23.5625488Z 146M /opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib 2021-04-05T11:30:23.5626989Z 146M /opt/conda/pkgs/valgrind-3.15.0-he513fc3_0 2021-04-05T11:30:23.5793611Z 349M /opt/conda/pkgs 2021-04-05T11:30:23.5845265Z 3.7G /opt/conda 2021-04-05T11:30:23.5846086Z 4.1G /opt 2021-04-05T11:30:23.6810817Z 131M /root/.m2/repository/org 2021-04-05T11:30:23.6813841Z 260M /root/.m2/repository 2021-04-05T11:30:23.6814253Z 260M /root/.m2 2021-04-05T11:30:23.7360947Z 124M /root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git/objects/pack 2021-04-05T11:30:23.7362421Z 124M /root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git/objects 2021-04-05T11:30:23.7376583Z 125M /root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git 2021-04-05T11:30:23.7447433Z 133M /root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823 2021-04-05T11:30:23.7448341Z 133M /root/.cargo/registry/index 2021-04-05T11:30:23.7448796Z 232M /root/.cargo/registry 2021-04-05T11:30:23.7449202Z 245M /root/.cargo 2021-04-05T11:30:23.8658627Z 126M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html/core 2021-04-05T11:30:23.8663924Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html 2021-04-05T11:30:23.8668130Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust 2021-04-05T11:30:23.8669570Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc 2021-04-05T11:30:23.8670941Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share 2021-04-05T11:30:23.8672364Z 149M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib 2021-04-05T11:30:23.8673637Z 155M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu 2021-04-05T11:30:23.8674758Z 157M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib 2021-04-05T11:30:23.8675721Z 376M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib 2021-04-05T11:30:23.8676657Z 865M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu 2021-04-05T11:30:23.8677417Z 865M /root/.rustup/toolchains 2021-04-05T11:30:23.8677825Z 865M /root/.rustup 2021-04-05T11:30:23.8678171Z 1.5G /root 2021-04-05T11:30:23.9219564Z 113M /go/pkg/mod/google.golang.org/[email protected]/googleapis 2021-04-05T11:30:23.9220667Z 113M /go/pkg/mod/google.golang.org/[email protected] 2021-04-05T11:30:23.9306825Z 129M /go/pkg/mod/google.golang.org 2021-04-05T11:30:23.9346436Z 264M /go/pkg/mod 2021-04-05T11:30:23.9346938Z 264M /go/pkg 2021-04-05T11:30:23.9347281Z 264M /go 2021-04-05T11:30:23.9383575Z 129M /build/cpp/src/arrow 2021-04-05T11:30:23.9384025Z 131M /build/cpp/src 2021-04-05T11:30:23.9387580Z 223M /build/cpp 2021-04-05T11:30:23.9387915Z 223M /build 2021-04-05T11:30:23.9388188Z 6.1G / 2021-04-05T11:30:24.0044084Z 846M /arrow/ ``` Closes apache#9889 from alamb/alamb/reduce_integration_size Authored-by: Andrew Lamb <[email protected]> Signed-off-by: Andrew Lamb <[email protected]>
alamb
added a commit
that referenced
this pull request
Apr 7, 2021
…tests # Rationale 1. As @jorgecarleitao noted on #9889 (comment), we should be running the check if arrow compiles with stable rust as that is what we target for the arrow crate 2. I noticed that there were several redundant (and inconsistent) settings of `RUSTFLAGS` 3. The titles of many of the tests are confusing (to me) as they have a lot of detailed architecture / rust version information before the description of what they are testing  # Changes 1. Use rust stable for the check that ensures the crate builds without default features 2. Remove redundant `RUSTFLAGS` 3. Change titles of the jobs to consistently start with a description of what they do # Note This could be three individual PRs, but I wanted to avoid the overhead of three separate JIRA tickets and juggling several concurrent potentially conflicting PRs. I will break it into three individual ones however if reviewers want. Closes #9904 from alamb/alamb/cleanup_rust_ci Authored-by: Andrew Lamb <[email protected]> Signed-off-by: Andrew Lamb <[email protected]>
alamb
added a commit
to apache/arrow-rs
that referenced
this pull request
Apr 20, 2021
…tests # Rationale 1. As @jorgecarleitao noted on apache/arrow#9889 (comment), we should be running the check if arrow compiles with stable rust as that is what we target for the arrow crate 2. I noticed that there were several redundant (and inconsistent) settings of `RUSTFLAGS` 3. The titles of many of the tests are confusing (to me) as they have a lot of detailed architecture / rust version information before the description of what they are testing  # Changes 1. Use rust stable for the check that ensures the crate builds without default features 2. Remove redundant `RUSTFLAGS` 3. Change titles of the jobs to consistently start with a description of what they do # Note This could be three individual PRs, but I wanted to avoid the overhead of three separate JIRA tickets and juggling several concurrent potentially conflicting PRs. I will break it into three individual ones however if reviewers want. Closes #9904 from alamb/alamb/cleanup_rust_ci Authored-by: Andrew Lamb <[email protected]> Signed-off-by: Andrew Lamb <[email protected]>
GeorgeAp
pushed a commit
to sirensolutions/arrow
that referenced
this pull request
Jun 7, 2021
…ation test # Rationale The [integration test](https://github.com/apache/arrow/pull/9884/checks?check_run_id=2263730460) has a fixed size builder docker image and has builds from several Arrow implementations. The Rust build artifacts (compiled binaries) in the integration tests still consume ~ 1GB of space even after apache#9879 (see @pitrou 's comment on apache#9884 (comment)). # Changes 1. Only build the `arrow-integration-testing` crate (which has the binaries needed for integration testing) rather than all of them 2. Remove incremental build artifacts 3. Moves `cargo check` to ensure we can build arrow without default features to a github action and out of the integration test This should both speed up the rust build as part of integration testing as well as reduce the amount of disk space required in the builder (both peak and after the rust build). # Results According to this [log](https://pipelines.actions.githubusercontent.com/CKhEcbzaHKPSiCoVWyWCFardoFMA9UN2zDnXYiBlRYO31o0IIv/_apis/pipelines/1/runs/179726/signedlogcontent/3?urlExpires=2021-04-05T12%3A37%3A13.1888704Z&urlSigningMethod=HMACV1&urlSignature=zROZZZham0BGH5f2vbe65n3DL13xXLgD9RJXybRqBAA%3D) after this PR's changes, the rust artifacts are much smaller. Note we could still save space by removing the rust toolchain and `.cargo` files but I think that can be an optimization for another time: ``` 2021-04-05T11:30:22.6164120Z 244M /opt/go/pkg 2021-04-05T11:30:22.6164599Z 393M /opt/go 2021-04-05T11:30:22.7201596Z 110M /opt/conda/envs/arrow/share 2021-04-05T11:30:22.7220002Z 171M /opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu/9.3.0 2021-04-05T11:30:22.7220911Z 171M /opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu 2021-04-05T11:30:22.7221489Z 171M /opt/conda/envs/arrow/libexec/gcc 2021-04-05T11:30:22.7221945Z 186M /opt/conda/envs/arrow/libexec 2021-04-05T11:30:22.7314210Z 186M /opt/conda/envs/arrow/lib/valgrind 2021-04-05T11:30:22.8109246Z 129M /opt/conda/envs/arrow/lib/python3.8/site-packages 2021-04-05T11:30:22.8290900Z 177M /opt/conda/envs/arrow/lib/python3.8 2021-04-05T11:30:22.8636030Z 1.8G /opt/conda/envs/arrow/lib 2021-04-05T11:30:22.9861104Z 171M /opt/conda/envs/arrow/include/boost 2021-04-05T11:30:23.0128341Z 260M /opt/conda/envs/arrow/include 2021-04-05T11:30:23.0163805Z 417M /opt/conda/envs/arrow/bin 2021-04-05T11:30:23.0449297Z 102M /opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr/lib64 2021-04-05T11:30:23.0525177Z 133M /opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr 2021-04-05T11:30:23.0526023Z 139M /opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot 2021-04-05T11:30:23.0527418Z 189M /opt/conda/envs/arrow/x86_64-conda-linux-gnu 2021-04-05T11:30:23.0612702Z 108M /opt/conda/envs/arrow/jre/lib 2021-04-05T11:30:23.0613140Z 109M /opt/conda/envs/arrow/jre 2021-04-05T11:30:23.0618402Z 3.2G /opt/conda/envs/arrow 2021-04-05T11:30:23.0618775Z 3.2G /opt/conda/envs 2021-04-05T11:30:23.1055113Z 159M /opt/conda/lib 2021-04-05T11:30:23.5619580Z 146M /opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib/valgrind 2021-04-05T11:30:23.5625488Z 146M /opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib 2021-04-05T11:30:23.5626989Z 146M /opt/conda/pkgs/valgrind-3.15.0-he513fc3_0 2021-04-05T11:30:23.5793611Z 349M /opt/conda/pkgs 2021-04-05T11:30:23.5845265Z 3.7G /opt/conda 2021-04-05T11:30:23.5846086Z 4.1G /opt 2021-04-05T11:30:23.6810817Z 131M /root/.m2/repository/org 2021-04-05T11:30:23.6813841Z 260M /root/.m2/repository 2021-04-05T11:30:23.6814253Z 260M /root/.m2 2021-04-05T11:30:23.7360947Z 124M /root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git/objects/pack 2021-04-05T11:30:23.7362421Z 124M /root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git/objects 2021-04-05T11:30:23.7376583Z 125M /root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git 2021-04-05T11:30:23.7447433Z 133M /root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823 2021-04-05T11:30:23.7448341Z 133M /root/.cargo/registry/index 2021-04-05T11:30:23.7448796Z 232M /root/.cargo/registry 2021-04-05T11:30:23.7449202Z 245M /root/.cargo 2021-04-05T11:30:23.8658627Z 126M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html/core 2021-04-05T11:30:23.8663924Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html 2021-04-05T11:30:23.8668130Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust 2021-04-05T11:30:23.8669570Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc 2021-04-05T11:30:23.8670941Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share 2021-04-05T11:30:23.8672364Z 149M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib 2021-04-05T11:30:23.8673637Z 155M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu 2021-04-05T11:30:23.8674758Z 157M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib 2021-04-05T11:30:23.8675721Z 376M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib 2021-04-05T11:30:23.8676657Z 865M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu 2021-04-05T11:30:23.8677417Z 865M /root/.rustup/toolchains 2021-04-05T11:30:23.8677825Z 865M /root/.rustup 2021-04-05T11:30:23.8678171Z 1.5G /root 2021-04-05T11:30:23.9219564Z 113M /go/pkg/mod/google.golang.org/[email protected]/googleapis 2021-04-05T11:30:23.9220667Z 113M /go/pkg/mod/google.golang.org/[email protected] 2021-04-05T11:30:23.9306825Z 129M /go/pkg/mod/google.golang.org 2021-04-05T11:30:23.9346436Z 264M /go/pkg/mod 2021-04-05T11:30:23.9346938Z 264M /go/pkg 2021-04-05T11:30:23.9347281Z 264M /go 2021-04-05T11:30:23.9383575Z 129M /build/cpp/src/arrow 2021-04-05T11:30:23.9384025Z 131M /build/cpp/src 2021-04-05T11:30:23.9387580Z 223M /build/cpp 2021-04-05T11:30:23.9387915Z 223M /build 2021-04-05T11:30:23.9388188Z 6.1G / 2021-04-05T11:30:24.0044084Z 846M /arrow/ ``` Closes apache#9889 from alamb/alamb/reduce_integration_size Authored-by: Andrew Lamb <[email protected]> Signed-off-by: Andrew Lamb <[email protected]>
GeorgeAp
pushed a commit
to sirensolutions/arrow
that referenced
this pull request
Jun 7, 2021
…tests # Rationale 1. As @jorgecarleitao noted on apache#9889 (comment), we should be running the check if arrow compiles with stable rust as that is what we target for the arrow crate 2. I noticed that there were several redundant (and inconsistent) settings of `RUSTFLAGS` 3. The titles of many of the tests are confusing (to me) as they have a lot of detailed architecture / rust version information before the description of what they are testing  # Changes 1. Use rust stable for the check that ensures the crate builds without default features 2. Remove redundant `RUSTFLAGS` 3. Change titles of the jobs to consistently start with a description of what they do # Note This could be three individual PRs, but I wanted to avoid the overhead of three separate JIRA tickets and juggling several concurrent potentially conflicting PRs. I will break it into three individual ones however if reviewers want. Closes apache#9904 from alamb/alamb/cleanup_rust_ci Authored-by: Andrew Lamb <[email protected]> Signed-off-by: Andrew Lamb <[email protected]>
michalursa
pushed a commit
to michalursa/arrow
that referenced
this pull request
Jun 10, 2021
…ation test # Rationale The [integration test](https://github.com/apache/arrow/pull/9884/checks?check_run_id=2263730460) has a fixed size builder docker image and has builds from several Arrow implementations. The Rust build artifacts (compiled binaries) in the integration tests still consume ~ 1GB of space even after apache#9879 (see @pitrou 's comment on apache#9884 (comment)). # Changes 1. Only build the `arrow-integration-testing` crate (which has the binaries needed for integration testing) rather than all of them 2. Remove incremental build artifacts 3. Moves `cargo check` to ensure we can build arrow without default features to a github action and out of the integration test This should both speed up the rust build as part of integration testing as well as reduce the amount of disk space required in the builder (both peak and after the rust build). # Results According to this [log](https://pipelines.actions.githubusercontent.com/CKhEcbzaHKPSiCoVWyWCFardoFMA9UN2zDnXYiBlRYO31o0IIv/_apis/pipelines/1/runs/179726/signedlogcontent/3?urlExpires=2021-04-05T12%3A37%3A13.1888704Z&urlSigningMethod=HMACV1&urlSignature=zROZZZham0BGH5f2vbe65n3DL13xXLgD9RJXybRqBAA%3D) after this PR's changes, the rust artifacts are much smaller. Note we could still save space by removing the rust toolchain and `.cargo` files but I think that can be an optimization for another time: ``` 2021-04-05T11:30:22.6164120Z 244M /opt/go/pkg 2021-04-05T11:30:22.6164599Z 393M /opt/go 2021-04-05T11:30:22.7201596Z 110M /opt/conda/envs/arrow/share 2021-04-05T11:30:22.7220002Z 171M /opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu/9.3.0 2021-04-05T11:30:22.7220911Z 171M /opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu 2021-04-05T11:30:22.7221489Z 171M /opt/conda/envs/arrow/libexec/gcc 2021-04-05T11:30:22.7221945Z 186M /opt/conda/envs/arrow/libexec 2021-04-05T11:30:22.7314210Z 186M /opt/conda/envs/arrow/lib/valgrind 2021-04-05T11:30:22.8109246Z 129M /opt/conda/envs/arrow/lib/python3.8/site-packages 2021-04-05T11:30:22.8290900Z 177M /opt/conda/envs/arrow/lib/python3.8 2021-04-05T11:30:22.8636030Z 1.8G /opt/conda/envs/arrow/lib 2021-04-05T11:30:22.9861104Z 171M /opt/conda/envs/arrow/include/boost 2021-04-05T11:30:23.0128341Z 260M /opt/conda/envs/arrow/include 2021-04-05T11:30:23.0163805Z 417M /opt/conda/envs/arrow/bin 2021-04-05T11:30:23.0449297Z 102M /opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr/lib64 2021-04-05T11:30:23.0525177Z 133M /opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr 2021-04-05T11:30:23.0526023Z 139M /opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot 2021-04-05T11:30:23.0527418Z 189M /opt/conda/envs/arrow/x86_64-conda-linux-gnu 2021-04-05T11:30:23.0612702Z 108M /opt/conda/envs/arrow/jre/lib 2021-04-05T11:30:23.0613140Z 109M /opt/conda/envs/arrow/jre 2021-04-05T11:30:23.0618402Z 3.2G /opt/conda/envs/arrow 2021-04-05T11:30:23.0618775Z 3.2G /opt/conda/envs 2021-04-05T11:30:23.1055113Z 159M /opt/conda/lib 2021-04-05T11:30:23.5619580Z 146M /opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib/valgrind 2021-04-05T11:30:23.5625488Z 146M /opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib 2021-04-05T11:30:23.5626989Z 146M /opt/conda/pkgs/valgrind-3.15.0-he513fc3_0 2021-04-05T11:30:23.5793611Z 349M /opt/conda/pkgs 2021-04-05T11:30:23.5845265Z 3.7G /opt/conda 2021-04-05T11:30:23.5846086Z 4.1G /opt 2021-04-05T11:30:23.6810817Z 131M /root/.m2/repository/org 2021-04-05T11:30:23.6813841Z 260M /root/.m2/repository 2021-04-05T11:30:23.6814253Z 260M /root/.m2 2021-04-05T11:30:23.7360947Z 124M /root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git/objects/pack 2021-04-05T11:30:23.7362421Z 124M /root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git/objects 2021-04-05T11:30:23.7376583Z 125M /root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git 2021-04-05T11:30:23.7447433Z 133M /root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823 2021-04-05T11:30:23.7448341Z 133M /root/.cargo/registry/index 2021-04-05T11:30:23.7448796Z 232M /root/.cargo/registry 2021-04-05T11:30:23.7449202Z 245M /root/.cargo 2021-04-05T11:30:23.8658627Z 126M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html/core 2021-04-05T11:30:23.8663924Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html 2021-04-05T11:30:23.8668130Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust 2021-04-05T11:30:23.8669570Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc 2021-04-05T11:30:23.8670941Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share 2021-04-05T11:30:23.8672364Z 149M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib 2021-04-05T11:30:23.8673637Z 155M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu 2021-04-05T11:30:23.8674758Z 157M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib 2021-04-05T11:30:23.8675721Z 376M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib 2021-04-05T11:30:23.8676657Z 865M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu 2021-04-05T11:30:23.8677417Z 865M /root/.rustup/toolchains 2021-04-05T11:30:23.8677825Z 865M /root/.rustup 2021-04-05T11:30:23.8678171Z 1.5G /root 2021-04-05T11:30:23.9219564Z 113M /go/pkg/mod/google.golang.org/[email protected]/googleapis 2021-04-05T11:30:23.9220667Z 113M /go/pkg/mod/google.golang.org/[email protected] 2021-04-05T11:30:23.9306825Z 129M /go/pkg/mod/google.golang.org 2021-04-05T11:30:23.9346436Z 264M /go/pkg/mod 2021-04-05T11:30:23.9346938Z 264M /go/pkg 2021-04-05T11:30:23.9347281Z 264M /go 2021-04-05T11:30:23.9383575Z 129M /build/cpp/src/arrow 2021-04-05T11:30:23.9384025Z 131M /build/cpp/src 2021-04-05T11:30:23.9387580Z 223M /build/cpp 2021-04-05T11:30:23.9387915Z 223M /build 2021-04-05T11:30:23.9388188Z 6.1G / 2021-04-05T11:30:24.0044084Z 846M /arrow/ ``` Closes apache#9889 from alamb/alamb/reduce_integration_size Authored-by: Andrew Lamb <[email protected]> Signed-off-by: Andrew Lamb <[email protected]>
michalursa
pushed a commit
to michalursa/arrow
that referenced
this pull request
Jun 10, 2021
…tests # Rationale 1. As @jorgecarleitao noted on apache#9889 (comment), we should be running the check if arrow compiles with stable rust as that is what we target for the arrow crate 2. I noticed that there were several redundant (and inconsistent) settings of `RUSTFLAGS` 3. The titles of many of the tests are confusing (to me) as they have a lot of detailed architecture / rust version information before the description of what they are testing  # Changes 1. Use rust stable for the check that ensures the crate builds without default features 2. Remove redundant `RUSTFLAGS` 3. Change titles of the jobs to consistently start with a description of what they do # Note This could be three individual PRs, but I wanted to avoid the overhead of three separate JIRA tickets and juggling several concurrent potentially conflicting PRs. I will break it into three individual ones however if reviewers want. Closes apache#9904 from alamb/alamb/cleanup_rust_ci Authored-by: Andrew Lamb <[email protected]> Signed-off-by: Andrew Lamb <[email protected]>
michalursa
pushed a commit
to michalursa/arrow
that referenced
this pull request
Jun 13, 2021
…ation test # Rationale The [integration test](https://github.com/apache/arrow/pull/9884/checks?check_run_id=2263730460) has a fixed size builder docker image and has builds from several Arrow implementations. The Rust build artifacts (compiled binaries) in the integration tests still consume ~ 1GB of space even after apache#9879 (see @pitrou 's comment on apache#9884 (comment)). # Changes 1. Only build the `arrow-integration-testing` crate (which has the binaries needed for integration testing) rather than all of them 2. Remove incremental build artifacts 3. Moves `cargo check` to ensure we can build arrow without default features to a github action and out of the integration test This should both speed up the rust build as part of integration testing as well as reduce the amount of disk space required in the builder (both peak and after the rust build). # Results According to this [log](https://pipelines.actions.githubusercontent.com/CKhEcbzaHKPSiCoVWyWCFardoFMA9UN2zDnXYiBlRYO31o0IIv/_apis/pipelines/1/runs/179726/signedlogcontent/3?urlExpires=2021-04-05T12%3A37%3A13.1888704Z&urlSigningMethod=HMACV1&urlSignature=zROZZZham0BGH5f2vbe65n3DL13xXLgD9RJXybRqBAA%3D) after this PR's changes, the rust artifacts are much smaller. Note we could still save space by removing the rust toolchain and `.cargo` files but I think that can be an optimization for another time: ``` 2021-04-05T11:30:22.6164120Z 244M /opt/go/pkg 2021-04-05T11:30:22.6164599Z 393M /opt/go 2021-04-05T11:30:22.7201596Z 110M /opt/conda/envs/arrow/share 2021-04-05T11:30:22.7220002Z 171M /opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu/9.3.0 2021-04-05T11:30:22.7220911Z 171M /opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu 2021-04-05T11:30:22.7221489Z 171M /opt/conda/envs/arrow/libexec/gcc 2021-04-05T11:30:22.7221945Z 186M /opt/conda/envs/arrow/libexec 2021-04-05T11:30:22.7314210Z 186M /opt/conda/envs/arrow/lib/valgrind 2021-04-05T11:30:22.8109246Z 129M /opt/conda/envs/arrow/lib/python3.8/site-packages 2021-04-05T11:30:22.8290900Z 177M /opt/conda/envs/arrow/lib/python3.8 2021-04-05T11:30:22.8636030Z 1.8G /opt/conda/envs/arrow/lib 2021-04-05T11:30:22.9861104Z 171M /opt/conda/envs/arrow/include/boost 2021-04-05T11:30:23.0128341Z 260M /opt/conda/envs/arrow/include 2021-04-05T11:30:23.0163805Z 417M /opt/conda/envs/arrow/bin 2021-04-05T11:30:23.0449297Z 102M /opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr/lib64 2021-04-05T11:30:23.0525177Z 133M /opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr 2021-04-05T11:30:23.0526023Z 139M /opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot 2021-04-05T11:30:23.0527418Z 189M /opt/conda/envs/arrow/x86_64-conda-linux-gnu 2021-04-05T11:30:23.0612702Z 108M /opt/conda/envs/arrow/jre/lib 2021-04-05T11:30:23.0613140Z 109M /opt/conda/envs/arrow/jre 2021-04-05T11:30:23.0618402Z 3.2G /opt/conda/envs/arrow 2021-04-05T11:30:23.0618775Z 3.2G /opt/conda/envs 2021-04-05T11:30:23.1055113Z 159M /opt/conda/lib 2021-04-05T11:30:23.5619580Z 146M /opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib/valgrind 2021-04-05T11:30:23.5625488Z 146M /opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib 2021-04-05T11:30:23.5626989Z 146M /opt/conda/pkgs/valgrind-3.15.0-he513fc3_0 2021-04-05T11:30:23.5793611Z 349M /opt/conda/pkgs 2021-04-05T11:30:23.5845265Z 3.7G /opt/conda 2021-04-05T11:30:23.5846086Z 4.1G /opt 2021-04-05T11:30:23.6810817Z 131M /root/.m2/repository/org 2021-04-05T11:30:23.6813841Z 260M /root/.m2/repository 2021-04-05T11:30:23.6814253Z 260M /root/.m2 2021-04-05T11:30:23.7360947Z 124M /root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git/objects/pack 2021-04-05T11:30:23.7362421Z 124M /root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git/objects 2021-04-05T11:30:23.7376583Z 125M /root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git 2021-04-05T11:30:23.7447433Z 133M /root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823 2021-04-05T11:30:23.7448341Z 133M /root/.cargo/registry/index 2021-04-05T11:30:23.7448796Z 232M /root/.cargo/registry 2021-04-05T11:30:23.7449202Z 245M /root/.cargo 2021-04-05T11:30:23.8658627Z 126M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html/core 2021-04-05T11:30:23.8663924Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html 2021-04-05T11:30:23.8668130Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust 2021-04-05T11:30:23.8669570Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc 2021-04-05T11:30:23.8670941Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share 2021-04-05T11:30:23.8672364Z 149M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib 2021-04-05T11:30:23.8673637Z 155M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu 2021-04-05T11:30:23.8674758Z 157M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib 2021-04-05T11:30:23.8675721Z 376M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib 2021-04-05T11:30:23.8676657Z 865M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu 2021-04-05T11:30:23.8677417Z 865M /root/.rustup/toolchains 2021-04-05T11:30:23.8677825Z 865M /root/.rustup 2021-04-05T11:30:23.8678171Z 1.5G /root 2021-04-05T11:30:23.9219564Z 113M /go/pkg/mod/google.golang.org/[email protected]/googleapis 2021-04-05T11:30:23.9220667Z 113M /go/pkg/mod/google.golang.org/[email protected] 2021-04-05T11:30:23.9306825Z 129M /go/pkg/mod/google.golang.org 2021-04-05T11:30:23.9346436Z 264M /go/pkg/mod 2021-04-05T11:30:23.9346938Z 264M /go/pkg 2021-04-05T11:30:23.9347281Z 264M /go 2021-04-05T11:30:23.9383575Z 129M /build/cpp/src/arrow 2021-04-05T11:30:23.9384025Z 131M /build/cpp/src 2021-04-05T11:30:23.9387580Z 223M /build/cpp 2021-04-05T11:30:23.9387915Z 223M /build 2021-04-05T11:30:23.9388188Z 6.1G / 2021-04-05T11:30:24.0044084Z 846M /arrow/ ``` Closes apache#9889 from alamb/alamb/reduce_integration_size Authored-by: Andrew Lamb <[email protected]> Signed-off-by: Andrew Lamb <[email protected]>
michalursa
pushed a commit
to michalursa/arrow
that referenced
this pull request
Jun 13, 2021
…tests # Rationale 1. As @jorgecarleitao noted on apache#9889 (comment), we should be running the check if arrow compiles with stable rust as that is what we target for the arrow crate 2. I noticed that there were several redundant (and inconsistent) settings of `RUSTFLAGS` 3. The titles of many of the tests are confusing (to me) as they have a lot of detailed architecture / rust version information before the description of what they are testing  # Changes 1. Use rust stable for the check that ensures the crate builds without default features 2. Remove redundant `RUSTFLAGS` 3. Change titles of the jobs to consistently start with a description of what they do # Note This could be three individual PRs, but I wanted to avoid the overhead of three separate JIRA tickets and juggling several concurrent potentially conflicting PRs. I will break it into three individual ones however if reviewers want. Closes apache#9904 from alamb/alamb/cleanup_rust_ci Authored-by: Andrew Lamb <[email protected]> Signed-off-by: Andrew Lamb <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rationale
The integration test has a fixed size builder docker image and has builds from several Arrow implementations.
The Rust build artifacts (compiled binaries) in the integration tests still consume ~ 1GB of space even after #9879 (see @pitrou 's comment on #9884 (comment)).
Changes
arrow-integration-testingcrate (which has the binaries needed for integration testing) rather than all of themcargo checkto ensure we can build arrow without default features to a github action and out of the integration testThis should both speed up the rust build as part of integration testing as well as reduce the amount of disk space required in the builder (both peak and after the rust build).
Results
According to this log after this PR's changes, the rust artifacts are much smaller.
Note we could still save space by removing the rust toolchain and
.cargofiles but I think that can be an optimization for another time: