categorize sandbox creation failure

[Skgland]

This is so that crater can categorize this as a spurious failure similar to SandboxOOM.

See rust-lang/crater#700 (comment)

[syphar]

Any chance of finding out if the test failures are related to the code changes?

[syphar]

It also would be nice to have a test, but I see that it's

1. probably not that easy
2. other error variants also don't have a specific test

[Skgland]

I ran the tests locally under WSL 2 in Windows 11

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy

there all tests, including the two failing here, pass.

Edit: though my toolchain is outdated

cargo --version
cargo 1.78.0 (54d8815d0 2024-03-26)

Edit: updated toolchain and tests still pass locally

cargo --version
cargo 1.83.0 (5ffbef321 2024-10-29)

[Skgland]

Attempting to run the test on my other machine runnign linux mint I can reproduce these errors, but I also get several more.

One thing to note is that I had to run cargo update as the version of openssl in the lockfile wouldn't compile otherwise.

/target/target/debug/deps/tests-75d57df3e11d0c5c: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /target/target/debug/deps/tests-75d57df3e11d0c5c)
/target/target/debug/deps/tests-75d57df3e11d0c5c: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.39' not found (required by /target/target/debug/deps/tests-75d57df3e11d0c5c)
test buildtest::inside_docker::test_hello_world ... FAILED
/target/target/debug/deps/tests-75d57df3e11d0c5c: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /target/target/debug/deps/tests-75d57df3e11d0c5c)
/target/target/debug/deps/tests-75d57df3e11d0c5c: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.39' not found (required by /target/target/debug/deps/tests-75d57df3e11d0c5c)
test buildtest::inside_docker::test_path_based_patch ... FAILED

I think the problem (with the tests failing here) might be that the tests are build outside of docker which has a recent version of glibc, but then run in docker with an older version of glibc.

Edit:

The docker image appears to have glibc 2.35

$ docker run -- ghcr.io/rust-lang/crates-build-env/linux-micro /lib/x86_64-linux-gnu/libc.so.6      
GNU C Library (Ubuntu GLIBC 2.35-0ubuntu3.8) stable release version 2.35.
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 11.4.0.
libc ABIs: UNIQUE IFUNC ABSOLUTE
For bug reporting instructions, please see:
<https://bugs.launchpad.net/ubuntu/+source/glibc/+bugs>.

The tests failing for me locally on linux mint are:

failures:
    buildtest::inside_docker::test_hello_world
    buildtest::inside_docker::test_path_based_patch
    buildtest::path_based_patch
    buildtest::test_cargo_workspace
    buildtest::test_hello_world
    buildtest::test_override_files
    buildtest::test_process_lines
    buildtest::test_sandbox_oom
    integration::purge_caches::test_purge_caches

test result: FAILED. 10 passed; 9 failed; 0 ignored; 0 measured; 0 filtered out; finished in 37.65s

I am currently still investigating.

For the tests that fail locally but not here in CI I get this error

[INFO  rustwide::cmd] running `Command { std: "docker" "start" "-a" "233e3b5cba16d3f9c74086bba779f5843759051d72b8a50639d354121e810442", kill_on_drop: false }`
[INFO  rustwide::cmd] [stderr] error: failed to create directory `/opt/rustwide/target/debug`
[INFO  rustwide::cmd] [stderr] 
[INFO  rustwide::cmd] [stderr] Caused by:
[INFO  rustwide::cmd] [stderr]   Permission denied (os error 13)

[Skgland]

Any chance of finding out if the test failures are related to the code changes?

I now made a checkout of the master branch and the same tests are failing so this appears to be unrelated to this PR.

failures:
    buildtest::inside_docker::test_hello_world
    buildtest::inside_docker::test_path_based_patch
    buildtest::path_based_patch
    buildtest::test_cargo_workspace
    buildtest::test_hello_world
    buildtest::test_override_files
    buildtest::test_process_lines
    buildtest::test_sandbox_oom
    integration::purge_caches::test_purge_caches

test result: FAILED. 10 passed; 9 failed; 0 ignored; 0 measured; 0 filtered out; finished in 36.85s

[Skgland]

It also would be nice to have a test, but I see that it's

1. probably not that easy

It might be possibel to setup a filesystem storage quota for docker and then have rustwide try to create a container that is expected to be larger than the quota, but I wouldn't know how to go about doing this.

2. other error variants also don't have a specific test

there is the buildtest::test_sandbox_oom test for SandboxOOM, though that simply has to set a low memory limit to force an OOM kill.

[Skgland]

Found the reason why my linux mint machine had more errors than CI,
apparently I had rootless docker configured and switching to normal docker resolved those extra errors,
leaving only the glibc mismatch.

[Skgland]

I think the source of the glibc error is that in the last CI run prior to this PR ubuntu-latest was 22.04.5

and for this PR its 24.04.1.

Which causes the tests to be build against a newer glibc version than is available in the crates-build-env docker image.

Solutions would be to either

1. to downgrade the image used to run tests to ubuntu-22
2. create an updated crates-build-end image with newer glibc (Update to Noble 24.04 crates-build-env#151)
3. somehow build against a sysroot with an older glibc

[syphar]

🤦 forgot the approval before merge :) So let's see this as approved.

@Skgland do you have more things to add? Otherwise I would create a release.

[Skgland]

No, no further additions from my side at the moment.

A few chores that could be done come to mind like updating/upgrading dependencies and rust edition, but nothing I require or want to do right now.

[syphar]

released in 0.19.0