Simple type-unstable code: 150x regression wrt 1.8

[aplavin]

See MWE below. Results are the same for beta3 and beta4.

# 1.8
julia> @btime NamedTuple{k[]}(v[])  setup=(k=Ref((:a,)); v=Ref((1,)))
  337.324 ns (6 allocations: 192 bytes)
(a = 1,)

# 1.9
julia> @btime NamedTuple{k[]}(v[])  setup=(k=Ref((:a,)); v=Ref((1,)))
  52.048 μs (44 allocations: 2.16 KiB)
(a = 1,)

Ref slack thread just in case: https://julialang.slack.com/archives/C67910KEH/p1675970292526539.

[Seelengrab]

For the record, I cannot reproduce on master:

julia> @btime NamedTuple{k[]}(v[])  setup=(k=Ref((:a,)); v=Ref((1,)))
  64.902 ns (2 allocations: 32 bytes)
(a = 1,)

julia> versioninfo()
Julia Version 1.10.0-DEV.546
Commit d72a9a1d2a (2023-02-09 17:01 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: 24 × AMD Ryzen 9 7900X 12-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, znver3)
  Threads: 24 on 24 virtual cores
Environment:
  JULIA_NUM_THREADS = 24

and also cannot reproduce on a 4 day old master:

julia> @btime NamedTuple{k[]}(v[])  setup=(k=Ref((:a,)); v=Ref((1,)))
  66.036 ns (2 allocations: 32 bytes)
(a = 1,)

julia> versioninfo()
Julia Version 1.10.0-DEV.496
Commit 07c4244caa (2023-02-05 01:47 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: 24 × AMD Ryzen 9 7900X 12-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, znver3)
  Threads: 24 on 24 virtual cores
Environment:
  JULIA_NUM_THREADS = 24

[vtjnash]

Sounds fixed then

[aplavin]

Fixed for 1.9, or only 1.10? I presume current master corresponds to the latter.

[t-bltg]

Sounds fixed then

It'd be nice to identify the PR fixing this on master, so that it can be backported to 1.9, no ?

[Seelengrab]

Sounds fixed then

However, I can reproduce on a local build of beta4:

julia> @btime NamedTuple{k[]}(v[])  setup=(k=Ref((:a,)); v=Ref((1,)))
  20.130 μs (45 allocations: 2.22 KiB)
(a = 1,)

julia> versioninfo()
Julia Version 1.9.0-beta4
Commit b75ddb787f (2023-02-07 21:53 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: 24 × AMD Ryzen 9 7900X 12-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, znver3)
  Threads: 24 on 24 virtual cores
Environment:
  JULIA_NUM_THREADS = 24

[martinholters]

Bisection led me to 1508425 (#48144) as the fix. Don't know whether that is backport-eligible.

[aplavin]

I tried the recent rc, and the regression is present there as well. Don't really understand the workflow in this repo: was the issue fixed and then broke again before rc? Or not fixed in any julia version in the first place? If so, where to track its actual status?

[KristofferC]

It is fixed on master but it hasn't been put into 1.9, see the discussion in #48144 (comment).

[aplavin]

So, this regression will be present in Julia during the whole 1.9 period, while considered "fixed"? TBH I'm personally quite concerned about this approach to 150x slowdowns, having usecases where performance is important.

[Seelengrab]

Not every change is backportable, even if it fixes a regression. There's just not enough hands to go around to fix every possible case, sadly enough. In this particular case it seems that the fix is due to a change deep in the compiler, possibly relying on other changes that happened since the branching of 1.9.

[aplavin]

Then I don't really get the point of "no breaking changes", if performance of simple code can drop by multiple orders of magnitude in a minor release, changing from useable to unusable in performance-sensitive code.
For me, in cursory testing of some code on 1.9, it went from "barely visible in the profiler graph" to "dominates the code runtime, making the whole computation 5x slower".

I (and potentially many other users?) would definitely prefer more stability, doesn't matter if it meant releases take more time and are made less frequently. Adding more and more new features (I don't argue with their usefulness) while not being able to rely on basic code working fast really seems weird.

[timholy]

Your code still runs and returns the same answers. If we weren't allowed to improve the compiler, Julia would be frozen which would be bad. But as you've noticed, some improvements come with corner-case regressions. Sorry. In the fullness of time we tend to get the benefits while eventually eliminating the harms, so the net progress is very positive.

The compiler team is stretched thin; unless you're proposing to do the work yourself, there are limits to what's possible, and going to heroics in a corner case takes away from progress that may be more impactful. You can continue to use Julia 1.8 until Julia 1.10 gets released.

[martinholters]

Also I'd like to point out that I've only bisected the performance recovery. Anyone willing to invest some time here could bisect what introduced the regression. That does not require much background knowledge but - with some luck - might point to an easier fix then backporting #48144.

[t-bltg]

I've bisected the regression to #45062.

On 36aab14, 192 bytes.
On 19f44b6, 2320 bytes.
We roughly have a 12x allocated bytes regression between 1.8.5 and 1.9.0-beta3.

The following sequence was used in the git bisect run script:

f(k, v) = NamedTuple{k[]}(v[])
k, v = Ref((:a,)), Ref((1,))
f(k, v)  # warmup
stats = @timed f(k, v)
print(stats.bytes)

I can't help notice that there only are a few allocations tests in the julia testsuite.
Should there be more of these tests (with an obvious tolerance for different OSs, ...) to catch these kind of regressions ?

[Seelengrab]

Most allocation benchmarks/statistics are tracked in https://github.com/JuliaCI/GCBenchmarks

[BioTurboNick]

Your code still runs and returns the same answers. If we weren't allowed to improve the compiler, Julia would be frozen which would be bad. But as you've noticed, some improvements come with corner-case regressions. Sorry. In the fullness of time we tend to get the benefits while eventually eliminating the harms, so the net progress is very positive.

The compiler team is stretched thin; unless you're proposing to do the work yourself, there are limits to what's possible, and going to heroics in a corner case takes away from progress that may be more impactful. You can continue to use Julia 1.8 until Julia 1.10 gets released.

Arguably not that much of a corner case if existing optimized code depends on the behavior. Though to be clear, for my case the overall code is still faster in 1.9, even if one critical part is slower.

At a minimum, this family of regressions should be documented in the release notes. With any workaround, if available.

It would be also nice to know if it is at all possible to hack in a patch for this behavior that isn't a full solution, just gets specific situations closer to normal.

Or if a warning could be emitted if this situation is detected, so users of 1.9 know what specifically is wrong without having to dig around.

If significant performance regressions are deemed necessary for a release, can we find ways to reduce the surprise factor?