Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ARROW-12112: [Rust][CI] Create and store less debug information in CI and integration tests #9879

Closed
wants to merge 1 commit into from
Closed

ARROW-12112: [Rust][CI] Create and store less debug information in CI and integration tests #9879

wants to merge 1 commit into from

Conversation

alamb
Copy link
Contributor

@alamb alamb commented Apr 2, 2021 •
edited
Loading

Rationale

Rust debug symbols are quite verbose, taking up memory during the final link time as well as significant disk space. Turning off the creation of symbols should save us compile / test time for CI as well as space on the integration test

Change

Do not produce debug symbols on Rust CI (keep enough to have line numbers in panic! traceback, but not enough to interpret a core file, which no one does to my knowledge anyways)

Note that the integration test passed: https://github.com/apache/arrow/pull/9879/checks?check_run_id=2256148363

@github-actions
Copy link

github-actions bot commented Apr 2, 2021

Thanks for opening a pull request!

Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

See also:

@alamb alamb force-pushed the less_symbols_in_integration branch from 683f51f to 04d0ad0 Compare April 2, 2021 19:39
@alamb alamb changed the title Less symbols in integration ARROW-TODO: [Rust] Less symbols in integration Apr 2, 2021
@alamb alamb changed the title ARROW-TODO: [Rust] Less symbols in integration ARROW-TODO: [Rust] Create and store less debug information in CI and integration tests Apr 2, 2021
@alamb
Copy link
Contributor Author

alamb commented Apr 2, 2021

I will do the CI / JIRA dance if this PR works out

@jacobBaumbach jacobBaumbach mentioned this pull request Apr 2, 2021
Closed
@alamb alamb changed the title ARROW-TODO: [Rust] Create and store less debug information in CI and integration tests ARROW-12112: [Rust] Create and store less debug information in CI and integration tests Apr 3, 2021
@github-actions
Copy link

github-actions bot commented Apr 3, 2021

https://issues.apache.org/jira/browse/ARROW-12112

jorgecarleitao
jorgecarleitao approved these changes Apr 3, 2021
View reviewed changes
Copy link
Member

@jorgecarleitao jorgecarleitao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks a lot for taking care of this! 👍

pitrou
pitrou approved these changes Apr 3, 2021
View reviewed changes
Copy link
Member

@pitrou pitrou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the record, how much space did this save?

@alamb
Copy link
Contributor Author

alamb commented Apr 3, 2021

For the record, how much space did this save?

@pitrou I did not measure it -- how did you do so?

@alamb alamb closed this in 2b87dfc Apr 3, 2021
@pitrou
Copy link
Member

pitrou commented Apr 3, 2021 •
edited
Loading

@alamb See here

@alamb alamb changed the title ARROW-12112: [Rust] Create and store less debug information in CI and integration tests ARROW-12112: [Rust][CI] Create and store less debug information in CI and integration tests Apr 3, 2021
This was referenced Apr 3, 2021
Closed
Closed
@alamb
Copy link
Contributor Author

alamb commented Apr 3, 2021

I have created #9883 and #9884 to find out

@alamb alamb mentioned this pull request Apr 4, 2021
Closed
ritchie46 pushed a commit to ritchie46/arrow that referenced this pull request Apr 5, 2021
@alamb @ritchie46
ARROW-12112: [Rust] Create and store less debug information in CI and…
b2ce4b6 
… integration tests

# Rationale
Rust debug symbols are quite verbose, taking up memory during the final link time as well as significant disk space. Turning off the creation of symbols should save us compile / test time for CI as well as space on the integration test

# Change
Do not produce debug symbols on Rust CI (keep enough to have line numbers in `panic!` traceback, but not enough to interpret a core file, which no one does to my knowledge anyways)

Note that the integration test passed: https://github.com/apache/arrow/pull/9879/checks?check_run_id=2256148363

Closes apache#9879 from alamb/less_symbols_in_integration

Authored-by: Andrew Lamb <[email protected]>
Signed-off-by: Andrew Lamb <[email protected]>
@alamb alamb mentioned this pull request Apr 5, 2021
Closed
pachadotdev pushed a commit to pachadotdev/arrow that referenced this pull request Apr 5, 2021
@alamb
ARROW-12112: [Rust] Create and store less debug information in CI and…
87844be 
… integration tests

# Rationale
Rust debug symbols are quite verbose, taking up memory during the final link time as well as significant disk space. Turning off the creation of symbols should save us compile / test time for CI as well as space on the integration test

# Change
Do not produce debug symbols on Rust CI (keep enough to have line numbers in `panic!` traceback, but not enough to interpret a core file, which no one does to my knowledge anyways)

Note that the integration test passed: https://github.com/apache/arrow/pull/9879/checks?check_run_id=2256148363

Closes apache#9879 from alamb/less_symbols_in_integration

Authored-by: Andrew Lamb <[email protected]>
Signed-off-by: Andrew Lamb <[email protected]>
alamb added a commit that referenced this pull request Apr 6, 2021
@alamb
ARROW-12204: [Rust][CI] Reduce size of Rust build artifacts in integr…
dd8cd10 
…ation test

# Rationale

The [integration test](https://github.com/apache/arrow/pull/9884/checks?check_run_id=2263730460) has a fixed size builder docker image and has builds from several Arrow implementations.

The Rust build artifacts (compiled binaries) in the integration tests still consume ~ 1GB of space even after #9879 (see @pitrou 's comment on  #9884 (comment)).

# Changes
1. Only build the `arrow-integration-testing` crate (which has the binaries needed for integration testing) rather than all of them
2. Remove incremental build artifacts
3. Moves `cargo check`  to ensure we can build arrow without default features to a github action and out of the integration test

This should both speed up the rust build as part of integration testing as well as reduce the amount of disk space required in the builder (both peak and after the rust build).

# Results

According to this [log](https://pipelines.actions.githubusercontent.com/CKhEcbzaHKPSiCoVWyWCFardoFMA9UN2zDnXYiBlRYO31o0IIv/_apis/pipelines/1/runs/179726/signedlogcontent/3?urlExpires=2021-04-05T12%3A37%3A13.1888704Z&urlSigningMethod=HMACV1&urlSignature=zROZZZham0BGH5f2vbe65n3DL13xXLgD9RJXybRqBAA%3D)  after this PR's changes, the rust artifacts are much smaller.

Note we could still save space by removing the rust toolchain and `.cargo` files but I think that can be an optimization for another time:
```
2021-04-05T11:30:22.6164120Z 244M	/opt/go/pkg
2021-04-05T11:30:22.6164599Z 393M	/opt/go
2021-04-05T11:30:22.7201596Z 110M	/opt/conda/envs/arrow/share
2021-04-05T11:30:22.7220002Z 171M	/opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu/9.3.0
2021-04-05T11:30:22.7220911Z 171M	/opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu
2021-04-05T11:30:22.7221489Z 171M	/opt/conda/envs/arrow/libexec/gcc
2021-04-05T11:30:22.7221945Z 186M	/opt/conda/envs/arrow/libexec
2021-04-05T11:30:22.7314210Z 186M	/opt/conda/envs/arrow/lib/valgrind
2021-04-05T11:30:22.8109246Z 129M	/opt/conda/envs/arrow/lib/python3.8/site-packages
2021-04-05T11:30:22.8290900Z 177M	/opt/conda/envs/arrow/lib/python3.8
2021-04-05T11:30:22.8636030Z 1.8G	/opt/conda/envs/arrow/lib
2021-04-05T11:30:22.9861104Z 171M	/opt/conda/envs/arrow/include/boost
2021-04-05T11:30:23.0128341Z 260M	/opt/conda/envs/arrow/include
2021-04-05T11:30:23.0163805Z 417M	/opt/conda/envs/arrow/bin
2021-04-05T11:30:23.0449297Z 102M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr/lib64
2021-04-05T11:30:23.0525177Z 133M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr
2021-04-05T11:30:23.0526023Z 139M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot
2021-04-05T11:30:23.0527418Z 189M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu
2021-04-05T11:30:23.0612702Z 108M	/opt/conda/envs/arrow/jre/lib
2021-04-05T11:30:23.0613140Z 109M	/opt/conda/envs/arrow/jre
2021-04-05T11:30:23.0618402Z 3.2G	/opt/conda/envs/arrow
2021-04-05T11:30:23.0618775Z 3.2G	/opt/conda/envs
2021-04-05T11:30:23.1055113Z 159M	/opt/conda/lib
2021-04-05T11:30:23.5619580Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib/valgrind
2021-04-05T11:30:23.5625488Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib
2021-04-05T11:30:23.5626989Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0
2021-04-05T11:30:23.5793611Z 349M	/opt/conda/pkgs
2021-04-05T11:30:23.5845265Z 3.7G	/opt/conda
2021-04-05T11:30:23.5846086Z 4.1G	/opt
2021-04-05T11:30:23.6810817Z 131M	/root/.m2/repository/org
2021-04-05T11:30:23.6813841Z 260M	/root/.m2/repository
2021-04-05T11:30:23.6814253Z 260M	/root/.m2
2021-04-05T11:30:23.7360947Z 124M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git/objects/pack
2021-04-05T11:30:23.7362421Z 124M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git/objects
2021-04-05T11:30:23.7376583Z 125M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git
2021-04-05T11:30:23.7447433Z 133M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823
2021-04-05T11:30:23.7448341Z 133M	/root/.cargo/registry/index
2021-04-05T11:30:23.7448796Z 232M	/root/.cargo/registry
2021-04-05T11:30:23.7449202Z 245M	/root/.cargo
2021-04-05T11:30:23.8658627Z 126M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html/core
2021-04-05T11:30:23.8663924Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html
2021-04-05T11:30:23.8668130Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust
2021-04-05T11:30:23.8669570Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc
2021-04-05T11:30:23.8670941Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share
2021-04-05T11:30:23.8672364Z 149M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib
2021-04-05T11:30:23.8673637Z 155M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu
2021-04-05T11:30:23.8674758Z 157M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib
2021-04-05T11:30:23.8675721Z 376M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib
2021-04-05T11:30:23.8676657Z 865M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu
2021-04-05T11:30:23.8677417Z 865M	/root/.rustup/toolchains
2021-04-05T11:30:23.8677825Z 865M	/root/.rustup
2021-04-05T11:30:23.8678171Z 1.5G	/root
2021-04-05T11:30:23.9219564Z 113M	/go/pkg/mod/google.golang.org/[email protected]/googleapis
2021-04-05T11:30:23.9220667Z 113M	/go/pkg/mod/google.golang.org/[email protected]
2021-04-05T11:30:23.9306825Z 129M	/go/pkg/mod/google.golang.org
2021-04-05T11:30:23.9346436Z 264M	/go/pkg/mod
2021-04-05T11:30:23.9346938Z 264M	/go/pkg
2021-04-05T11:30:23.9347281Z 264M	/go
2021-04-05T11:30:23.9383575Z 129M	/build/cpp/src/arrow
2021-04-05T11:30:23.9384025Z 131M	/build/cpp/src
2021-04-05T11:30:23.9387580Z 223M	/build/cpp
2021-04-05T11:30:23.9387915Z 223M	/build
2021-04-05T11:30:23.9388188Z 6.1G	/
2021-04-05T11:30:24.0044084Z 846M	/arrow/
```

Closes #9889 from alamb/alamb/reduce_integration_size

Authored-by: Andrew Lamb <[email protected]>
Signed-off-by: Andrew Lamb <[email protected]>
pachadotdev pushed a commit to pachadotdev/arrow that referenced this pull request Apr 6, 2021
@alamb
ARROW-12204: [Rust][CI] Reduce size of Rust build artifacts in integr…
4e41c48 
…ation test

# Rationale

The [integration test](https://github.com/apache/arrow/pull/9884/checks?check_run_id=2263730460) has a fixed size builder docker image and has builds from several Arrow implementations.

The Rust build artifacts (compiled binaries) in the integration tests still consume ~ 1GB of space even after apache#9879 (see @pitrou 's comment on  apache#9884 (comment)).

# Changes
1. Only build the `arrow-integration-testing` crate (which has the binaries needed for integration testing) rather than all of them
2. Remove incremental build artifacts
3. Moves `cargo check`  to ensure we can build arrow without default features to a github action and out of the integration test

This should both speed up the rust build as part of integration testing as well as reduce the amount of disk space required in the builder (both peak and after the rust build).

# Results

According to this [log](https://pipelines.actions.githubusercontent.com/CKhEcbzaHKPSiCoVWyWCFardoFMA9UN2zDnXYiBlRYO31o0IIv/_apis/pipelines/1/runs/179726/signedlogcontent/3?urlExpires=2021-04-05T12%3A37%3A13.1888704Z&urlSigningMethod=HMACV1&urlSignature=zROZZZham0BGH5f2vbe65n3DL13xXLgD9RJXybRqBAA%3D)  after this PR's changes, the rust artifacts are much smaller.

Note we could still save space by removing the rust toolchain and `.cargo` files but I think that can be an optimization for another time:
```
2021-04-05T11:30:22.6164120Z 244M	/opt/go/pkg
2021-04-05T11:30:22.6164599Z 393M	/opt/go
2021-04-05T11:30:22.7201596Z 110M	/opt/conda/envs/arrow/share
2021-04-05T11:30:22.7220002Z 171M	/opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu/9.3.0
2021-04-05T11:30:22.7220911Z 171M	/opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu
2021-04-05T11:30:22.7221489Z 171M	/opt/conda/envs/arrow/libexec/gcc
2021-04-05T11:30:22.7221945Z 186M	/opt/conda/envs/arrow/libexec
2021-04-05T11:30:22.7314210Z 186M	/opt/conda/envs/arrow/lib/valgrind
2021-04-05T11:30:22.8109246Z 129M	/opt/conda/envs/arrow/lib/python3.8/site-packages
2021-04-05T11:30:22.8290900Z 177M	/opt/conda/envs/arrow/lib/python3.8
2021-04-05T11:30:22.8636030Z 1.8G	/opt/conda/envs/arrow/lib
2021-04-05T11:30:22.9861104Z 171M	/opt/conda/envs/arrow/include/boost
2021-04-05T11:30:23.0128341Z 260M	/opt/conda/envs/arrow/include
2021-04-05T11:30:23.0163805Z 417M	/opt/conda/envs/arrow/bin
2021-04-05T11:30:23.0449297Z 102M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr/lib64
2021-04-05T11:30:23.0525177Z 133M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr
2021-04-05T11:30:23.0526023Z 139M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot
2021-04-05T11:30:23.0527418Z 189M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu
2021-04-05T11:30:23.0612702Z 108M	/opt/conda/envs/arrow/jre/lib
2021-04-05T11:30:23.0613140Z 109M	/opt/conda/envs/arrow/jre
2021-04-05T11:30:23.0618402Z 3.2G	/opt/conda/envs/arrow
2021-04-05T11:30:23.0618775Z 3.2G	/opt/conda/envs
2021-04-05T11:30:23.1055113Z 159M	/opt/conda/lib
2021-04-05T11:30:23.5619580Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib/valgrind
2021-04-05T11:30:23.5625488Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib
2021-04-05T11:30:23.5626989Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0
2021-04-05T11:30:23.5793611Z 349M	/opt/conda/pkgs
2021-04-05T11:30:23.5845265Z 3.7G	/opt/conda
2021-04-05T11:30:23.5846086Z 4.1G	/opt
2021-04-05T11:30:23.6810817Z 131M	/root/.m2/repository/org
2021-04-05T11:30:23.6813841Z 260M	/root/.m2/repository
2021-04-05T11:30:23.6814253Z 260M	/root/.m2
2021-04-05T11:30:23.7360947Z 124M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git/objects/pack
2021-04-05T11:30:23.7362421Z 124M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git/objects
2021-04-05T11:30:23.7376583Z 125M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git
2021-04-05T11:30:23.7447433Z 133M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823
2021-04-05T11:30:23.7448341Z 133M	/root/.cargo/registry/index
2021-04-05T11:30:23.7448796Z 232M	/root/.cargo/registry
2021-04-05T11:30:23.7449202Z 245M	/root/.cargo
2021-04-05T11:30:23.8658627Z 126M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html/core
2021-04-05T11:30:23.8663924Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html
2021-04-05T11:30:23.8668130Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust
2021-04-05T11:30:23.8669570Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc
2021-04-05T11:30:23.8670941Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share
2021-04-05T11:30:23.8672364Z 149M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib
2021-04-05T11:30:23.8673637Z 155M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu
2021-04-05T11:30:23.8674758Z 157M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib
2021-04-05T11:30:23.8675721Z 376M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib
2021-04-05T11:30:23.8676657Z 865M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu
2021-04-05T11:30:23.8677417Z 865M	/root/.rustup/toolchains
2021-04-05T11:30:23.8677825Z 865M	/root/.rustup
2021-04-05T11:30:23.8678171Z 1.5G	/root
2021-04-05T11:30:23.9219564Z 113M	/go/pkg/mod/google.golang.org/[email protected]/googleapis
2021-04-05T11:30:23.9220667Z 113M	/go/pkg/mod/google.golang.org/[email protected]
2021-04-05T11:30:23.9306825Z 129M	/go/pkg/mod/google.golang.org
2021-04-05T11:30:23.9346436Z 264M	/go/pkg/mod
2021-04-05T11:30:23.9346938Z 264M	/go/pkg
2021-04-05T11:30:23.9347281Z 264M	/go
2021-04-05T11:30:23.9383575Z 129M	/build/cpp/src/arrow
2021-04-05T11:30:23.9384025Z 131M	/build/cpp/src
2021-04-05T11:30:23.9387580Z 223M	/build/cpp
2021-04-05T11:30:23.9387915Z 223M	/build
2021-04-05T11:30:23.9388188Z 6.1G	/
2021-04-05T11:30:24.0044084Z 846M	/arrow/
```

Closes apache#9889 from alamb/alamb/reduce_integration_size

Authored-by: Andrew Lamb <[email protected]>
Signed-off-by: Andrew Lamb <[email protected]>
alamb added a commit to apache/arrow-rs that referenced this pull request Apr 20, 2021
@alamb
ARROW-12204: [Rust][CI] Reduce size of Rust build artifacts in integr…
9337e10 
…ation test

# Rationale

The [integration test](https://github.com/apache/arrow/pull/9884/checks?check_run_id=2263730460) has a fixed size builder docker image and has builds from several Arrow implementations.

The Rust build artifacts (compiled binaries) in the integration tests still consume ~ 1GB of space even after apache/arrow#9879 (see @pitrou 's comment on  apache/arrow#9884 (comment)).

# Changes
1. Only build the `arrow-integration-testing` crate (which has the binaries needed for integration testing) rather than all of them
2. Remove incremental build artifacts
3. Moves `cargo check`  to ensure we can build arrow without default features to a github action and out of the integration test

This should both speed up the rust build as part of integration testing as well as reduce the amount of disk space required in the builder (both peak and after the rust build).

# Results

According to this [log](https://pipelines.actions.githubusercontent.com/CKhEcbzaHKPSiCoVWyWCFardoFMA9UN2zDnXYiBlRYO31o0IIv/_apis/pipelines/1/runs/179726/signedlogcontent/3?urlExpires=2021-04-05T12%3A37%3A13.1888704Z&urlSigningMethod=HMACV1&urlSignature=zROZZZham0BGH5f2vbe65n3DL13xXLgD9RJXybRqBAA%3D)  after this PR's changes, the rust artifacts are much smaller.

Note we could still save space by removing the rust toolchain and `.cargo` files but I think that can be an optimization for another time:
```
2021-04-05T11:30:22.6164120Z 244M	/opt/go/pkg
2021-04-05T11:30:22.6164599Z 393M	/opt/go
2021-04-05T11:30:22.7201596Z 110M	/opt/conda/envs/arrow/share
2021-04-05T11:30:22.7220002Z 171M	/opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu/9.3.0
2021-04-05T11:30:22.7220911Z 171M	/opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu
2021-04-05T11:30:22.7221489Z 171M	/opt/conda/envs/arrow/libexec/gcc
2021-04-05T11:30:22.7221945Z 186M	/opt/conda/envs/arrow/libexec
2021-04-05T11:30:22.7314210Z 186M	/opt/conda/envs/arrow/lib/valgrind
2021-04-05T11:30:22.8109246Z 129M	/opt/conda/envs/arrow/lib/python3.8/site-packages
2021-04-05T11:30:22.8290900Z 177M	/opt/conda/envs/arrow/lib/python3.8
2021-04-05T11:30:22.8636030Z 1.8G	/opt/conda/envs/arrow/lib
2021-04-05T11:30:22.9861104Z 171M	/opt/conda/envs/arrow/include/boost
2021-04-05T11:30:23.0128341Z 260M	/opt/conda/envs/arrow/include
2021-04-05T11:30:23.0163805Z 417M	/opt/conda/envs/arrow/bin
2021-04-05T11:30:23.0449297Z 102M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr/lib64
2021-04-05T11:30:23.0525177Z 133M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr
2021-04-05T11:30:23.0526023Z 139M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot
2021-04-05T11:30:23.0527418Z 189M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu
2021-04-05T11:30:23.0612702Z 108M	/opt/conda/envs/arrow/jre/lib
2021-04-05T11:30:23.0613140Z 109M	/opt/conda/envs/arrow/jre
2021-04-05T11:30:23.0618402Z 3.2G	/opt/conda/envs/arrow
2021-04-05T11:30:23.0618775Z 3.2G	/opt/conda/envs
2021-04-05T11:30:23.1055113Z 159M	/opt/conda/lib
2021-04-05T11:30:23.5619580Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib/valgrind
2021-04-05T11:30:23.5625488Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib
2021-04-05T11:30:23.5626989Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0
2021-04-05T11:30:23.5793611Z 349M	/opt/conda/pkgs
2021-04-05T11:30:23.5845265Z 3.7G	/opt/conda
2021-04-05T11:30:23.5846086Z 4.1G	/opt
2021-04-05T11:30:23.6810817Z 131M	/root/.m2/repository/org
2021-04-05T11:30:23.6813841Z 260M	/root/.m2/repository
2021-04-05T11:30:23.6814253Z 260M	/root/.m2
2021-04-05T11:30:23.7360947Z 124M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git/objects/pack
2021-04-05T11:30:23.7362421Z 124M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git/objects
2021-04-05T11:30:23.7376583Z 125M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git
2021-04-05T11:30:23.7447433Z 133M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823
2021-04-05T11:30:23.7448341Z 133M	/root/.cargo/registry/index
2021-04-05T11:30:23.7448796Z 232M	/root/.cargo/registry
2021-04-05T11:30:23.7449202Z 245M	/root/.cargo
2021-04-05T11:30:23.8658627Z 126M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html/core
2021-04-05T11:30:23.8663924Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html
2021-04-05T11:30:23.8668130Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust
2021-04-05T11:30:23.8669570Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc
2021-04-05T11:30:23.8670941Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share
2021-04-05T11:30:23.8672364Z 149M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib
2021-04-05T11:30:23.8673637Z 155M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu
2021-04-05T11:30:23.8674758Z 157M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib
2021-04-05T11:30:23.8675721Z 376M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib
2021-04-05T11:30:23.8676657Z 865M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu
2021-04-05T11:30:23.8677417Z 865M	/root/.rustup/toolchains
2021-04-05T11:30:23.8677825Z 865M	/root/.rustup
2021-04-05T11:30:23.8678171Z 1.5G	/root
2021-04-05T11:30:23.9219564Z 113M	/go/pkg/mod/google.golang.org/[email protected]/googleapis
2021-04-05T11:30:23.9220667Z 113M	/go/pkg/mod/google.golang.org/[email protected]
2021-04-05T11:30:23.9306825Z 129M	/go/pkg/mod/google.golang.org
2021-04-05T11:30:23.9346436Z 264M	/go/pkg/mod
2021-04-05T11:30:23.9346938Z 264M	/go/pkg
2021-04-05T11:30:23.9347281Z 264M	/go
2021-04-05T11:30:23.9383575Z 129M	/build/cpp/src/arrow
2021-04-05T11:30:23.9384025Z 131M	/build/cpp/src
2021-04-05T11:30:23.9387580Z 223M	/build/cpp
2021-04-05T11:30:23.9387915Z 223M	/build
2021-04-05T11:30:23.9388188Z 6.1G	/
2021-04-05T11:30:24.0044084Z 846M	/arrow/
```

Closes #9889 from alamb/alamb/reduce_integration_size

Authored-by: Andrew Lamb <[email protected]>
Signed-off-by: Andrew Lamb <[email protected]>
GeorgeAp pushed a commit to sirensolutions/arrow that referenced this pull request Jun 7, 2021
@alamb @GeorgeAp
ARROW-12112: [Rust] Create and store less debug information in CI and…
9067e04 
… integration tests

# Rationale
Rust debug symbols are quite verbose, taking up memory during the final link time as well as significant disk space. Turning off the creation of symbols should save us compile / test time for CI as well as space on the integration test

# Change
Do not produce debug symbols on Rust CI (keep enough to have line numbers in `panic!` traceback, but not enough to interpret a core file, which no one does to my knowledge anyways)

Note that the integration test passed: https://github.com/apache/arrow/pull/9879/checks?check_run_id=2256148363

Closes apache#9879 from alamb/less_symbols_in_integration

Authored-by: Andrew Lamb <[email protected]>
Signed-off-by: Andrew Lamb <[email protected]>
GeorgeAp pushed a commit to sirensolutions/arrow that referenced this pull request Jun 7, 2021
@alamb @GeorgeAp
ARROW-12204: [Rust][CI] Reduce size of Rust build artifacts in integr…
04b4971 
…ation test

# Rationale

The [integration test](https://github.com/apache/arrow/pull/9884/checks?check_run_id=2263730460) has a fixed size builder docker image and has builds from several Arrow implementations.

The Rust build artifacts (compiled binaries) in the integration tests still consume ~ 1GB of space even after apache#9879 (see @pitrou 's comment on  apache#9884 (comment)).

# Changes
1. Only build the `arrow-integration-testing` crate (which has the binaries needed for integration testing) rather than all of them
2. Remove incremental build artifacts
3. Moves `cargo check`  to ensure we can build arrow without default features to a github action and out of the integration test

This should both speed up the rust build as part of integration testing as well as reduce the amount of disk space required in the builder (both peak and after the rust build).

# Results

According to this [log](https://pipelines.actions.githubusercontent.com/CKhEcbzaHKPSiCoVWyWCFardoFMA9UN2zDnXYiBlRYO31o0IIv/_apis/pipelines/1/runs/179726/signedlogcontent/3?urlExpires=2021-04-05T12%3A37%3A13.1888704Z&urlSigningMethod=HMACV1&urlSignature=zROZZZham0BGH5f2vbe65n3DL13xXLgD9RJXybRqBAA%3D)  after this PR's changes, the rust artifacts are much smaller.

Note we could still save space by removing the rust toolchain and `.cargo` files but I think that can be an optimization for another time:
```
2021-04-05T11:30:22.6164120Z 244M	/opt/go/pkg
2021-04-05T11:30:22.6164599Z 393M	/opt/go
2021-04-05T11:30:22.7201596Z 110M	/opt/conda/envs/arrow/share
2021-04-05T11:30:22.7220002Z 171M	/opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu/9.3.0
2021-04-05T11:30:22.7220911Z 171M	/opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu
2021-04-05T11:30:22.7221489Z 171M	/opt/conda/envs/arrow/libexec/gcc
2021-04-05T11:30:22.7221945Z 186M	/opt/conda/envs/arrow/libexec
2021-04-05T11:30:22.7314210Z 186M	/opt/conda/envs/arrow/lib/valgrind
2021-04-05T11:30:22.8109246Z 129M	/opt/conda/envs/arrow/lib/python3.8/site-packages
2021-04-05T11:30:22.8290900Z 177M	/opt/conda/envs/arrow/lib/python3.8
2021-04-05T11:30:22.8636030Z 1.8G	/opt/conda/envs/arrow/lib
2021-04-05T11:30:22.9861104Z 171M	/opt/conda/envs/arrow/include/boost
2021-04-05T11:30:23.0128341Z 260M	/opt/conda/envs/arrow/include
2021-04-05T11:30:23.0163805Z 417M	/opt/conda/envs/arrow/bin
2021-04-05T11:30:23.0449297Z 102M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr/lib64
2021-04-05T11:30:23.0525177Z 133M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr
2021-04-05T11:30:23.0526023Z 139M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot
2021-04-05T11:30:23.0527418Z 189M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu
2021-04-05T11:30:23.0612702Z 108M	/opt/conda/envs/arrow/jre/lib
2021-04-05T11:30:23.0613140Z 109M	/opt/conda/envs/arrow/jre
2021-04-05T11:30:23.0618402Z 3.2G	/opt/conda/envs/arrow
2021-04-05T11:30:23.0618775Z 3.2G	/opt/conda/envs
2021-04-05T11:30:23.1055113Z 159M	/opt/conda/lib
2021-04-05T11:30:23.5619580Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib/valgrind
2021-04-05T11:30:23.5625488Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib
2021-04-05T11:30:23.5626989Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0
2021-04-05T11:30:23.5793611Z 349M	/opt/conda/pkgs
2021-04-05T11:30:23.5845265Z 3.7G	/opt/conda
2021-04-05T11:30:23.5846086Z 4.1G	/opt
2021-04-05T11:30:23.6810817Z 131M	/root/.m2/repository/org
2021-04-05T11:30:23.6813841Z 260M	/root/.m2/repository
2021-04-05T11:30:23.6814253Z 260M	/root/.m2
2021-04-05T11:30:23.7360947Z 124M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git/objects/pack
2021-04-05T11:30:23.7362421Z 124M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git/objects
2021-04-05T11:30:23.7376583Z 125M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git
2021-04-05T11:30:23.7447433Z 133M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823
2021-04-05T11:30:23.7448341Z 133M	/root/.cargo/registry/index
2021-04-05T11:30:23.7448796Z 232M	/root/.cargo/registry
2021-04-05T11:30:23.7449202Z 245M	/root/.cargo
2021-04-05T11:30:23.8658627Z 126M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html/core
2021-04-05T11:30:23.8663924Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html
2021-04-05T11:30:23.8668130Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust
2021-04-05T11:30:23.8669570Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc
2021-04-05T11:30:23.8670941Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share
2021-04-05T11:30:23.8672364Z 149M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib
2021-04-05T11:30:23.8673637Z 155M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu
2021-04-05T11:30:23.8674758Z 157M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib
2021-04-05T11:30:23.8675721Z 376M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib
2021-04-05T11:30:23.8676657Z 865M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu
2021-04-05T11:30:23.8677417Z 865M	/root/.rustup/toolchains
2021-04-05T11:30:23.8677825Z 865M	/root/.rustup
2021-04-05T11:30:23.8678171Z 1.5G	/root
2021-04-05T11:30:23.9219564Z 113M	/go/pkg/mod/google.golang.org/[email protected]/googleapis
2021-04-05T11:30:23.9220667Z 113M	/go/pkg/mod/google.golang.org/[email protected]
2021-04-05T11:30:23.9306825Z 129M	/go/pkg/mod/google.golang.org
2021-04-05T11:30:23.9346436Z 264M	/go/pkg/mod
2021-04-05T11:30:23.9346938Z 264M	/go/pkg
2021-04-05T11:30:23.9347281Z 264M	/go
2021-04-05T11:30:23.9383575Z 129M	/build/cpp/src/arrow
2021-04-05T11:30:23.9384025Z 131M	/build/cpp/src
2021-04-05T11:30:23.9387580Z 223M	/build/cpp
2021-04-05T11:30:23.9387915Z 223M	/build
2021-04-05T11:30:23.9388188Z 6.1G	/
2021-04-05T11:30:24.0044084Z 846M	/arrow/
```

Closes apache#9889 from alamb/alamb/reduce_integration_size

Authored-by: Andrew Lamb <[email protected]>
Signed-off-by: Andrew Lamb <[email protected]>
michalursa pushed a commit to michalursa/arrow that referenced this pull request Jun 10, 2021
@alamb @michalursa
ARROW-12112: [Rust] Create and store less debug information in CI and…
16a3a04 
… integration tests

# Rationale
Rust debug symbols are quite verbose, taking up memory during the final link time as well as significant disk space. Turning off the creation of symbols should save us compile / test time for CI as well as space on the integration test

# Change
Do not produce debug symbols on Rust CI (keep enough to have line numbers in `panic!` traceback, but not enough to interpret a core file, which no one does to my knowledge anyways)

Note that the integration test passed: https://github.com/apache/arrow/pull/9879/checks?check_run_id=2256148363

Closes apache#9879 from alamb/less_symbols_in_integration

Authored-by: Andrew Lamb <[email protected]>
Signed-off-by: Andrew Lamb <[email protected]>
michalursa pushed a commit to michalursa/arrow that referenced this pull request Jun 10, 2021
@alamb @michalursa
ARROW-12204: [Rust][CI] Reduce size of Rust build artifacts in integr…
5fbdb89 
…ation test

# Rationale

The [integration test](https://github.com/apache/arrow/pull/9884/checks?check_run_id=2263730460) has a fixed size builder docker image and has builds from several Arrow implementations.

The Rust build artifacts (compiled binaries) in the integration tests still consume ~ 1GB of space even after apache#9879 (see @pitrou 's comment on  apache#9884 (comment)).

# Changes
1. Only build the `arrow-integration-testing` crate (which has the binaries needed for integration testing) rather than all of them
2. Remove incremental build artifacts
3. Moves `cargo check`  to ensure we can build arrow without default features to a github action and out of the integration test

This should both speed up the rust build as part of integration testing as well as reduce the amount of disk space required in the builder (both peak and after the rust build).

# Results

According to this [log](https://pipelines.actions.githubusercontent.com/CKhEcbzaHKPSiCoVWyWCFardoFMA9UN2zDnXYiBlRYO31o0IIv/_apis/pipelines/1/runs/179726/signedlogcontent/3?urlExpires=2021-04-05T12%3A37%3A13.1888704Z&urlSigningMethod=HMACV1&urlSignature=zROZZZham0BGH5f2vbe65n3DL13xXLgD9RJXybRqBAA%3D)  after this PR's changes, the rust artifacts are much smaller.

Note we could still save space by removing the rust toolchain and `.cargo` files but I think that can be an optimization for another time:
```
2021-04-05T11:30:22.6164120Z 244M	/opt/go/pkg
2021-04-05T11:30:22.6164599Z 393M	/opt/go
2021-04-05T11:30:22.7201596Z 110M	/opt/conda/envs/arrow/share
2021-04-05T11:30:22.7220002Z 171M	/opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu/9.3.0
2021-04-05T11:30:22.7220911Z 171M	/opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu
2021-04-05T11:30:22.7221489Z 171M	/opt/conda/envs/arrow/libexec/gcc
2021-04-05T11:30:22.7221945Z 186M	/opt/conda/envs/arrow/libexec
2021-04-05T11:30:22.7314210Z 186M	/opt/conda/envs/arrow/lib/valgrind
2021-04-05T11:30:22.8109246Z 129M	/opt/conda/envs/arrow/lib/python3.8/site-packages
2021-04-05T11:30:22.8290900Z 177M	/opt/conda/envs/arrow/lib/python3.8
2021-04-05T11:30:22.8636030Z 1.8G	/opt/conda/envs/arrow/lib
2021-04-05T11:30:22.9861104Z 171M	/opt/conda/envs/arrow/include/boost
2021-04-05T11:30:23.0128341Z 260M	/opt/conda/envs/arrow/include
2021-04-05T11:30:23.0163805Z 417M	/opt/conda/envs/arrow/bin
2021-04-05T11:30:23.0449297Z 102M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr/lib64
2021-04-05T11:30:23.0525177Z 133M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr
2021-04-05T11:30:23.0526023Z 139M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot
2021-04-05T11:30:23.0527418Z 189M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu
2021-04-05T11:30:23.0612702Z 108M	/opt/conda/envs/arrow/jre/lib
2021-04-05T11:30:23.0613140Z 109M	/opt/conda/envs/arrow/jre
2021-04-05T11:30:23.0618402Z 3.2G	/opt/conda/envs/arrow
2021-04-05T11:30:23.0618775Z 3.2G	/opt/conda/envs
2021-04-05T11:30:23.1055113Z 159M	/opt/conda/lib
2021-04-05T11:30:23.5619580Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib/valgrind
2021-04-05T11:30:23.5625488Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib
2021-04-05T11:30:23.5626989Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0
2021-04-05T11:30:23.5793611Z 349M	/opt/conda/pkgs
2021-04-05T11:30:23.5845265Z 3.7G	/opt/conda
2021-04-05T11:30:23.5846086Z 4.1G	/opt
2021-04-05T11:30:23.6810817Z 131M	/root/.m2/repository/org
2021-04-05T11:30:23.6813841Z 260M	/root/.m2/repository
2021-04-05T11:30:23.6814253Z 260M	/root/.m2
2021-04-05T11:30:23.7360947Z 124M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git/objects/pack
2021-04-05T11:30:23.7362421Z 124M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git/objects
2021-04-05T11:30:23.7376583Z 125M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git
2021-04-05T11:30:23.7447433Z 133M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823
2021-04-05T11:30:23.7448341Z 133M	/root/.cargo/registry/index
2021-04-05T11:30:23.7448796Z 232M	/root/.cargo/registry
2021-04-05T11:30:23.7449202Z 245M	/root/.cargo
2021-04-05T11:30:23.8658627Z 126M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html/core
2021-04-05T11:30:23.8663924Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html
2021-04-05T11:30:23.8668130Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust
2021-04-05T11:30:23.8669570Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc
2021-04-05T11:30:23.8670941Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share
2021-04-05T11:30:23.8672364Z 149M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib
2021-04-05T11:30:23.8673637Z 155M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu
2021-04-05T11:30:23.8674758Z 157M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib
2021-04-05T11:30:23.8675721Z 376M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib
2021-04-05T11:30:23.8676657Z 865M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu
2021-04-05T11:30:23.8677417Z 865M	/root/.rustup/toolchains
2021-04-05T11:30:23.8677825Z 865M	/root/.rustup
2021-04-05T11:30:23.8678171Z 1.5G	/root
2021-04-05T11:30:23.9219564Z 113M	/go/pkg/mod/google.golang.org/[email protected]/googleapis
2021-04-05T11:30:23.9220667Z 113M	/go/pkg/mod/google.golang.org/[email protected]
2021-04-05T11:30:23.9306825Z 129M	/go/pkg/mod/google.golang.org
2021-04-05T11:30:23.9346436Z 264M	/go/pkg/mod
2021-04-05T11:30:23.9346938Z 264M	/go/pkg
2021-04-05T11:30:23.9347281Z 264M	/go
2021-04-05T11:30:23.9383575Z 129M	/build/cpp/src/arrow
2021-04-05T11:30:23.9384025Z 131M	/build/cpp/src
2021-04-05T11:30:23.9387580Z 223M	/build/cpp
2021-04-05T11:30:23.9387915Z 223M	/build
2021-04-05T11:30:23.9388188Z 6.1G	/
2021-04-05T11:30:24.0044084Z 846M	/arrow/
```

Closes apache#9889 from alamb/alamb/reduce_integration_size

Authored-by: Andrew Lamb <[email protected]>
Signed-off-by: Andrew Lamb <[email protected]>
michalursa pushed a commit to michalursa/arrow that referenced this pull request Jun 13, 2021
@alamb @michalursa
ARROW-12112: [Rust] Create and store less debug information in CI and…
4184536 
… integration tests

# Rationale
Rust debug symbols are quite verbose, taking up memory during the final link time as well as significant disk space. Turning off the creation of symbols should save us compile / test time for CI as well as space on the integration test

# Change
Do not produce debug symbols on Rust CI (keep enough to have line numbers in `panic!` traceback, but not enough to interpret a core file, which no one does to my knowledge anyways)

Note that the integration test passed: https://github.com/apache/arrow/pull/9879/checks?check_run_id=2256148363

Closes apache#9879 from alamb/less_symbols_in_integration

Authored-by: Andrew Lamb <[email protected]>
Signed-off-by: Andrew Lamb <[email protected]>
michalursa pushed a commit to michalursa/arrow that referenced this pull request Jun 13, 2021
@alamb @michalursa
ARROW-12204: [Rust][CI] Reduce size of Rust build artifacts in integr…
14b8520 
…ation test

# Rationale

The [integration test](https://github.com/apache/arrow/pull/9884/checks?check_run_id=2263730460) has a fixed size builder docker image and has builds from several Arrow implementations.

The Rust build artifacts (compiled binaries) in the integration tests still consume ~ 1GB of space even after apache#9879 (see @pitrou 's comment on  apache#9884 (comment)).

# Changes
1. Only build the `arrow-integration-testing` crate (which has the binaries needed for integration testing) rather than all of them
2. Remove incremental build artifacts
3. Moves `cargo check`  to ensure we can build arrow without default features to a github action and out of the integration test

This should both speed up the rust build as part of integration testing as well as reduce the amount of disk space required in the builder (both peak and after the rust build).

# Results

According to this [log](https://pipelines.actions.githubusercontent.com/CKhEcbzaHKPSiCoVWyWCFardoFMA9UN2zDnXYiBlRYO31o0IIv/_apis/pipelines/1/runs/179726/signedlogcontent/3?urlExpires=2021-04-05T12%3A37%3A13.1888704Z&urlSigningMethod=HMACV1&urlSignature=zROZZZham0BGH5f2vbe65n3DL13xXLgD9RJXybRqBAA%3D)  after this PR's changes, the rust artifacts are much smaller.

Note we could still save space by removing the rust toolchain and `.cargo` files but I think that can be an optimization for another time:
```
2021-04-05T11:30:22.6164120Z 244M	/opt/go/pkg
2021-04-05T11:30:22.6164599Z 393M	/opt/go
2021-04-05T11:30:22.7201596Z 110M	/opt/conda/envs/arrow/share
2021-04-05T11:30:22.7220002Z 171M	/opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu/9.3.0
2021-04-05T11:30:22.7220911Z 171M	/opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu
2021-04-05T11:30:22.7221489Z 171M	/opt/conda/envs/arrow/libexec/gcc
2021-04-05T11:30:22.7221945Z 186M	/opt/conda/envs/arrow/libexec
2021-04-05T11:30:22.7314210Z 186M	/opt/conda/envs/arrow/lib/valgrind
2021-04-05T11:30:22.8109246Z 129M	/opt/conda/envs/arrow/lib/python3.8/site-packages
2021-04-05T11:30:22.8290900Z 177M	/opt/conda/envs/arrow/lib/python3.8
2021-04-05T11:30:22.8636030Z 1.8G	/opt/conda/envs/arrow/lib
2021-04-05T11:30:22.9861104Z 171M	/opt/conda/envs/arrow/include/boost
2021-04-05T11:30:23.0128341Z 260M	/opt/conda/envs/arrow/include
2021-04-05T11:30:23.0163805Z 417M	/opt/conda/envs/arrow/bin
2021-04-05T11:30:23.0449297Z 102M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr/lib64
2021-04-05T11:30:23.0525177Z 133M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr
2021-04-05T11:30:23.0526023Z 139M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot
2021-04-05T11:30:23.0527418Z 189M	/opt/conda/envs/arrow/x86_64-conda-linux-gnu
2021-04-05T11:30:23.0612702Z 108M	/opt/conda/envs/arrow/jre/lib
2021-04-05T11:30:23.0613140Z 109M	/opt/conda/envs/arrow/jre
2021-04-05T11:30:23.0618402Z 3.2G	/opt/conda/envs/arrow
2021-04-05T11:30:23.0618775Z 3.2G	/opt/conda/envs
2021-04-05T11:30:23.1055113Z 159M	/opt/conda/lib
2021-04-05T11:30:23.5619580Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib/valgrind
2021-04-05T11:30:23.5625488Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib
2021-04-05T11:30:23.5626989Z 146M	/opt/conda/pkgs/valgrind-3.15.0-he513fc3_0
2021-04-05T11:30:23.5793611Z 349M	/opt/conda/pkgs
2021-04-05T11:30:23.5845265Z 3.7G	/opt/conda
2021-04-05T11:30:23.5846086Z 4.1G	/opt
2021-04-05T11:30:23.6810817Z 131M	/root/.m2/repository/org
2021-04-05T11:30:23.6813841Z 260M	/root/.m2/repository
2021-04-05T11:30:23.6814253Z 260M	/root/.m2
2021-04-05T11:30:23.7360947Z 124M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git/objects/pack
2021-04-05T11:30:23.7362421Z 124M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git/objects
2021-04-05T11:30:23.7376583Z 125M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823/.git
2021-04-05T11:30:23.7447433Z 133M	/root/.cargo/registry/index/jackfan.us.kg-1ecc6299db9ec823
2021-04-05T11:30:23.7448341Z 133M	/root/.cargo/registry/index
2021-04-05T11:30:23.7448796Z 232M	/root/.cargo/registry
2021-04-05T11:30:23.7449202Z 245M	/root/.cargo
2021-04-05T11:30:23.8658627Z 126M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html/core
2021-04-05T11:30:23.8663924Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html
2021-04-05T11:30:23.8668130Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust
2021-04-05T11:30:23.8669570Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc
2021-04-05T11:30:23.8670941Z 423M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share
2021-04-05T11:30:23.8672364Z 149M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib
2021-04-05T11:30:23.8673637Z 155M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu
2021-04-05T11:30:23.8674758Z 157M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib
2021-04-05T11:30:23.8675721Z 376M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib
2021-04-05T11:30:23.8676657Z 865M	/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu
2021-04-05T11:30:23.8677417Z 865M	/root/.rustup/toolchains
2021-04-05T11:30:23.8677825Z 865M	/root/.rustup
2021-04-05T11:30:23.8678171Z 1.5G	/root
2021-04-05T11:30:23.9219564Z 113M	/go/pkg/mod/google.golang.org/[email protected]/googleapis
2021-04-05T11:30:23.9220667Z 113M	/go/pkg/mod/google.golang.org/[email protected]
2021-04-05T11:30:23.9306825Z 129M	/go/pkg/mod/google.golang.org
2021-04-05T11:30:23.9346436Z 264M	/go/pkg/mod
2021-04-05T11:30:23.9346938Z 264M	/go/pkg
2021-04-05T11:30:23.9347281Z 264M	/go
2021-04-05T11:30:23.9383575Z 129M	/build/cpp/src/arrow
2021-04-05T11:30:23.9384025Z 131M	/build/cpp/src
2021-04-05T11:30:23.9387580Z 223M	/build/cpp
2021-04-05T11:30:23.9387915Z 223M	/build
2021-04-05T11:30:23.9388188Z 6.1G	/
2021-04-05T11:30:24.0044084Z 846M	/arrow/
```

Closes apache#9889 from alamb/alamb/reduce_integration_size

Authored-by: Andrew Lamb <[email protected]>
Signed-off-by: Andrew Lamb <[email protected]>
This was referenced Apr 7, 2021
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jorgecarleitao jorgecarleitao jorgecarleitao approved these changes

@pitrou pitrou pitrou approved these changes

@nevi-me nevi-me Awaiting requested review from nevi-me

Assignees
No one assigned
Labels
Component: Rust - DataFusion Component: Rust
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@alamb @pitrou @jorgecarleitao