Only produce .xz tarballs on CI #80435

pietroalbini · 2020-12-28T12:34:32Z

This PR adds a ./configure option to choose which tarball compression formats to produce, and changes our CI configuration to only produce .xz tarballs. The release process will then recompress everything into .gz when producing a release.

This will drastically reduce our storage costs for CI artifacts, as we'd stop storing the same data twice. Stable, beta and nightly releases will not be affected by this at all.

Before landing this we'll need to increase the VM size of our release process, to recompress everything in a reasonable amount of time.
r? @Mark-Simulacrum

rust-highfive · 2020-12-28T12:34:35Z

⚠️ Warning ⚠️

These commits modify submodules.

Mark-Simulacrum

r=me with nit fixed

Mark-Simulacrum · 2020-12-29T02:13:00Z

src/bootstrap/tarball.rs

+            .as_ref()
+            .and_then(|formats| formats.get(0))
+            .map(|s| s.as_str())
+            .unwrap_or("gz");


hm, I think we should probably panic if the list is defined but empty, that seems like someone is doing the wrong thing.

Added an assert to ensure the list is not empty.

The option allows to add or remove compression formats used while producing dist tarballs.

pietroalbini · 2020-12-29T11:13:30Z

@bors r=Mark-Simulacrum

bors · 2020-12-29T11:13:32Z

📌 Commit d0d0ee0 has been approved by Mark-Simulacrum

…=Mark-Simulacrum Only produce .xz tarballs on CI This PR adds a `./configure` option to choose which tarball compression formats to produce, and changes our CI configuration to only produce `.xz` tarballs. The release process will then recompress everything into `.gz` when producing a release. This will drastically reduce our storage costs for CI artifacts, as we'd stop storing the same data twice. **Stable, beta and nightly releases will not be affected by this at all.** Before landing this we'll need to increase the VM size of our release process, to recompress everything in a reasonable amount of time. r? `@Mark-Simulacrum`

m-ou-se · 2020-12-30T13:38:52Z

This failed in #80496: https://github.com/rust-lang-ci/rust/runs/1624327931

gzip: stdin: not in gzip format

tar: Child returned status 1
tar: Error is not recoverable: exiting now

command did not execute successfully: "tar" "-xzf" "/checkout/obj/build/dist/rustc-nightly-src.tar.xz" "--strip-components=1"
expected success, got: exit code: 2

@bors r-

pietroalbini · 2020-12-30T15:06:59Z

@bors r=Mark-Simulacrum rollup=never

bors · 2020-12-30T15:07:00Z

📌 Commit f6b8572d1cdad991d030d5b07492d7410a749293 has been approved by Mark-Simulacrum

rust-log-analyzer · 2020-12-30T15:18:11Z

The job mingw-check failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

configure: build.locked-deps    := True
configure: build.cargo-native-static := True
configure: rust.codegen-units-std := 1
configure: rust.verify-llvm-ir  := True
configure: dist.compression-formats := ['xz']
configure: rust.channel         := nightly
configure: rust.debug-assertions := True
configure: llvm.assertions      := True
configure: dist.missing-tools   := True
configure: dist.missing-tools   := True
configure: build.configure-args := ['--enable-sccache', '--disable-manage-submodu ...
configure: writing `config.toml` in current directory
configure: 
configure: run `python /checkout/x.py --help`
configure: 
---
skip untracked path cpu-usage.csv during rustfmt invocations
skip untracked path src/doc/book/ during rustfmt invocations
skip untracked path src/doc/rust-by-example/ during rustfmt invocations
skip untracked path src/llvm-project/ during rustfmt invocations
Diff in /checkout/src/bootstrap/test.rs at line 1986:
         t!(fs::create_dir_all(&dir));
 
         let mut cmd = Command::new("tar");
-        cmd.arg("-xf")
-            .arg(builder.ensure(dist::Src))
-            .arg("--strip-components=1")
-            .current_dir(&dir);
+        cmd.arg("-xf").arg(builder.ensure(dist::Src)).arg("--strip-components=1").current_dir(&dir);
         builder.run(&mut cmd);
 
Running `"/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/rustfmt" "--config-path" "/checkout" "--edition" "2018" "--unstable-features" "--skip-children" "--check" "/checkout/src/bootstrap/test.rs"` failed.
If you're running `tidy`, try again with `--bless`. Or, if you just want to format code, run `./x.py fmt` instead.
         let toml = dir.join("rust-src/lib/rustlib/src/rust/library/std/Cargo.toml");
Build completed unsuccessfully in 0:00:17

pietroalbini · 2020-12-30T15:20:52Z

@bors r=Mark-Simulacrum rollup=never

bors · 2020-12-30T15:20:53Z

📌 Commit 5526d90 has been approved by Mark-Simulacrum

bors · 2020-12-30T18:24:31Z

⌛ Testing commit 5526d90 with merge 32f233eb3ad76bda1e72b3d92cfe57e8e0068cde...

rust-log-analyzer · 2020-12-30T18:30:41Z

The job dist-android failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

configure: target.x86_64-linux-android.android-ndk := /android/ndk/x86_64-21
configure: target.thumbv7neon-linux-androideabi.android-ndk := /android/ndk/arm-1 ...
configure: rust.debuginfo-level-std := 1
configure: llvm.static-libstdcpp := True
configure: dist.compression-formats := ['xz']
configure: build.locked-deps    := True
configure: build.profiler       := True
configure: target.aarch64-linux-android.android-ndk := /android/ndk/arm64-21
configure: rust.dist-src        := False
---
[TIMING] StdLink { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target_compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } -- 0.001
[TIMING] Std { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } } -- 90.635
Building LLVM for x86_64-unknown-linux-gnu
running: "cmake" "/checkout/src/llvm-project/llvm" "-G" "Ninja" "-DLLVM_ENABLE_ASSERTIONS=OFF" "-DLLVM_TARGETS_TO_BUILD=AArch64;ARM;Hexagon;MSP430;Mips;NVPTX;PowerPC;RISCV;Sparc;SystemZ;WebAssembly;X86" "-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=AVR" "-DLLVM_INCLUDE_EXAMPLES=OFF" "-DLLVM_INCLUDE_TESTS=OFF" "-DLLVM_INCLUDE_DOCS=OFF" "-DLLVM_INCLUDE_BENCHMARKS=OFF" "-DLLVM_ENABLE_TERMINFO=OFF" "-DLLVM_ENABLE_LIBEDIT=OFF" "-DLLVM_ENABLE_BINDINGS=OFF" "-DLLVM_ENABLE_Z3_SOLVER=OFF" "-DLLVM_PARALLEL_COMPILE_JOBS=16" "-DLLVM_TARGET_ARCH=x86_64" "-DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu" "-DLLVM_ENABLE_ZLIB=ON" "-DLLVM_ENABLE_LIBXML2=OFF" "-DLLVM_VERSION_SUFFIX=-rust-1.51.0-nightly" "-DCMAKE_INSTALL_MESSAGE=LAZY" "-DCMAKE_C_COMPILER_LAUNCHER=sccache" "-DCMAKE_CXX_COMPILER_LAUNCHER=sccache" "-DCMAKE_C_COMPILER=cc" "-DCMAKE_CXX_COMPILER=c++" "-DCMAKE_ASM_COMPILER=cc" "-DCMAKE_C_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_CXX_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64 -static-libstdc++" "-DCMAKE_INSTALL_PREFIX=/checkout/obj/build/x86_64-unknown-linux-gnu/llvm" "-DCMAKE_ASM_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_BUILD_TYPE=Release"
CMake Error: The source directory "/checkout/src/llvm-project/llvm" does not exist.
Specify --help for usage, or press the help button on the CMake GUI.
command did not execute successfully, got: exit code: 1

 finished in 0.011 seconds
 finished in 0.011 seconds
build script failed, must exit now', /cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/cmake-0.1.44/src/lib.rs:885:5
failed to run: /checkout/obj/build/bootstrap/debug/bootstrap dist --host= --target arm-linux-androideabi,armv7-linux-androideabi,thumbv7neon-linux-androideabi,i686-linux-android,aarch64-linux-android,x86_64-linux-android
Build completed unsuccessfully in 0:01:31

bors · 2020-12-30T18:32:25Z

💔 Test failed - checks-actions

Mark-Simulacrum · 2020-12-30T19:20:39Z

@bors retry - submodule download failed

bors · 2020-12-31T00:23:19Z

⌛ Testing commit 5526d90 with merge f0073a5...

bors · 2020-12-31T03:17:39Z

☀️ Test successful - checks-actions
Approved by: Mark-Simulacrum
Pushing f0073a5 to master...

Give a better error for download-ci-llvm if .xz is not supported Previously: ``` curl: (22) The requested URL returned error: 404 failed to run: curl -# -y 30 -Y 10 --connect-timeout 30 --retry 3 -Sf -o /tmp/tmp6ptXJV https://ci-artifacts.rust-lang.org/rustc-builds/99ad5a1a2824fea1ecf60068fd3636beae7ea2da/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.gz ``` Now: ``` error: XZ support is required to download LLVM help: consider disabling `download-ci-llvm` or using a different version of python Build completed unsuccessfully in 0:00:00 ``` Follow-up to rust-lang#80435. r? `@Mark-Simulacrum`

bump rust-installer submodule

c482ffd

rust-highfive assigned Mark-Simulacrum Dec 28, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Dec 28, 2020

Mark-Simulacrum reviewed Dec 29, 2020

View reviewed changes

Mark-Simulacrum added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 29, 2020

pietroalbini added 2 commits December 29, 2020 11:56

bootstrap: add the dist.compression-formats option

0a2034d

The option allows to add or remove compression formats used while producing dist tarballs.

ci: stop producing gzip-compressed dist tarballs

d0d0ee0

pietroalbini force-pushed the compression-formats branch from 78fe658 to d0d0ee0 Compare December 29, 2020 11:11

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Dec 29, 2020

Dylan-DPC-zz mentioned this pull request Dec 30, 2020

Rollup of 10 pull requests #80496

Closed

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Dec 30, 2020

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Dec 30, 2020

bootstrap: extract from any compression algorithm during distcheck

5526d90

pietroalbini force-pushed the compression-formats branch from f6b8572 to 5526d90 Compare December 30, 2020 15:20

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Dec 30, 2020

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 30, 2020

bors added the merged-by-bors This PR was explicitly merged by bors. label Dec 31, 2020

bors merged commit f0073a5 into rust-lang:master Dec 31, 2020

rustbot added this to the 1.51.0 milestone Dec 31, 2020

This was referenced Dec 31, 2020

Fix broken ./x.py install #80514

Merged

Rollup of 8 pull requests #80544

Closed

jyn514 mentioned this pull request Jan 1, 2021

Give a better error for download-ci-llvm if .xz is not supported #80581

Merged

pietroalbini mentioned this pull request Jan 6, 2021

Tracking issue for only producing .xz on CI #80753

Closed

6 tasks

pietroalbini deleted the compression-formats branch March 1, 2021 18:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Only produce .xz tarballs on CI #80435

Only produce .xz tarballs on CI #80435

pietroalbini commented Dec 28, 2020

rust-highfive commented Dec 28, 2020

Mark-Simulacrum left a comment

Mark-Simulacrum Dec 29, 2020

pietroalbini Dec 29, 2020

pietroalbini commented Dec 29, 2020

bors commented Dec 29, 2020

m-ou-se commented Dec 30, 2020

pietroalbini commented Dec 30, 2020

bors commented Dec 30, 2020

rust-log-analyzer commented Dec 30, 2020

pietroalbini commented Dec 30, 2020

bors commented Dec 30, 2020

bors commented Dec 30, 2020

rust-log-analyzer commented Dec 30, 2020

bors commented Dec 30, 2020

Mark-Simulacrum commented Dec 30, 2020

bors commented Dec 31, 2020

bors commented Dec 31, 2020

Only produce .xz tarballs on CI #80435

Only produce .xz tarballs on CI #80435

Conversation

pietroalbini commented Dec 28, 2020

rust-highfive commented Dec 28, 2020

Mark-Simulacrum left a comment

Choose a reason for hiding this comment

Mark-Simulacrum Dec 29, 2020

Choose a reason for hiding this comment

pietroalbini Dec 29, 2020

Choose a reason for hiding this comment

pietroalbini commented Dec 29, 2020

bors commented Dec 29, 2020

m-ou-se commented Dec 30, 2020

pietroalbini commented Dec 30, 2020

bors commented Dec 30, 2020

rust-log-analyzer commented Dec 30, 2020

pietroalbini commented Dec 30, 2020

bors commented Dec 30, 2020

bors commented Dec 30, 2020

rust-log-analyzer commented Dec 30, 2020

bors commented Dec 30, 2020

Mark-Simulacrum commented Dec 30, 2020

bors commented Dec 31, 2020

bors commented Dec 31, 2020