Base.runtests("core") fails on x86-64 macOS with 1.7.0

[carlocab]

While updating Homebrew's Julia to 1.7.0 (cf. Homebrew/homebrew-core#90194), I found that running Base.runtests("core") fails with

  Error in testset core:
  Test Failed at /usr/local/Cellar/julia/1.7.0/share/julia/test/core.jl:3519
    Expression: (Vararg{Int, N} where N <: T) where T
      Expected: ErrorException
    No exception thrown
  Error in testset core:
  Test Failed at /usr/local/Cellar/julia/1.7.0/share/julia/test/core.jl:3520
    Expression: Vararg{Int, N} where N <: Integer
      Expected: ErrorException
    No exception thrown
  Error in testset core:
  Test Failed at /usr/local/Cellar/julia/1.7.0/share/julia/test/core.jl:3521
    Expression: Vararg{Int, N} where N >: Integer
      Expected: ErrorException
    No exception thrown
  ERROR: LoadError: Test run finished with errors
  in expression starting at /usr/local/Cellar/julia/1.7.0/share/julia/test/runtests.jl:93
  ERROR: A test has failed. Please submit a bug report (https://github.com/JuliaLang/julia/issues)
  including error messages above and the output of versioninfo():
  Julia Version 1.7.0
  Commit 3bf9d17731 (2021-11-30 12:12 UTC)
  Platform Info:
    OS: macOS (x86_64-apple-darwin21.1.0)
    CPU: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz
    WORD_SIZE: 64
    LIBM: libopenlibm
    LLVM: libLLVM-12.0.1 (ORCJIT, ivybridge)

This reproduces on macOS 11 and 12. CI logs available here: https://github.com/Homebrew/homebrew-core/actions/runs/1522862535

To make sure that this isn't something peculiar to our build, I downloaded the official binary and encountered the exact same failure. Here's versioninfo() from the machine I ran that on:

Julia Version 1.7.0
Commit 3bf9d17731 (2021-11-30 12:12 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin19.5.0)
  CPU: Intel(R) Core(TM) i3-1000NG4 CPU @ 1.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, icelake-client)

Is it just a matter of these tests being outdated?

julia/test/core.jl

Lines 3519 to 3521 in 3bf9d17

	@test_throws ErrorException Vararg{Int, N} where N<:T where T
	@test_throws ErrorException Vararg{Int, N} where N<:Integer
	@test_throws ErrorException Vararg{Int, N} where N>:Integer

Those tests are still there on master, so I am wondering why this doesn't trip in CI here.

[t-bltg]

Same failure occurs on Linux with official binaries:

Julia Version 1.7.0
Commit 3bf9d17731 (2021-11-30 12:12 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, sandybridge)

[t-bltg]

However, running multiple tests makes core pass:
$ julia -e 'Base.runtests("int core")'

Test  (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB)
int        (2) |        started at 2021-12-01T12:59:19.463
core       (3) |        started at 2021-12-01T12:59:20.066
int        (2) |    17.50 |   0.26 |  1.5 |     750.38 |   354.79
core       (3) |   117.84 |  11.54 |  9.8 |   13383.00 |  2138.17

Test Summary: |    Pass  Broken    Total
  Overall     | 8970575       3  8970578
    SUCCESS

[martinholters]

Can't find issue or PR right now, but we recently stumbled upon those in some other context. The problem is that Vararg{Int, N} where N <: T is deprecated and either throws an error or not depending on CLI arguments, which may differ depending how tests are run. Is any case, those tests no longer test what they where desinged for.

[martinholters]

Aha, found it: duplicate of #43004.