Update performance tips to be explicit about compilation time practices

[ChrisRackauckas]

One of the things that has come up recently is that some benchmarks will include compilation time because "it's how I run code" or "it's a subjective thing", "it's a personal choice". The purpose of this change it to make it very explicit that this is not a personal choice and it is, as per the manual, using the programming language incorrectly. This gets rid of any debate around the topic and makes it easy for anyone, say a reviewer, to quote the manual in a way that says that any of the issues are user-induced because they are simply not programming or measuring in a style that mirrors common or recommended Julia usage.

[timholy]

I like the idea but this feels a bit heavy for what it's trying to convey, while also a bit disconnected from the rest of the performance tips page. What about incorporating this wisdom into the @time section? That section mentions discarding the first run, but perhaps we should just grab the bull by the horns and illustrate what we mean directly: (EDIT: ah, that second already includes a demo of the first time, I missed that when I first scanned it.)

blah blah...

julia> using LinearAlgebra

julia> @time eigen([2 1; 1 4])
  0.019775 seconds (35.04 k allocations: 2.221 MiB)
Eigen{Float64, Float64, Matrix{Float64}, Vector{Float64}}
values:
2-element Vector{Float64}:
 1.585786437626905
 4.414213562373095
vectors:
2×2 Matrix{Float64}:
 -0.92388   0.382683
  0.382683  0.92388

If you're experienced with scientific computing, you probably realize that 20 milliseconds seems like a very long time to compute the eigenvalues of a 2×2 matrix. The key point to realize, though, is that this includes the time required to compile eigen; if we run it a second time in the same session,

julia> @time eigen([2 1; 1 4])
  0.000042 seconds (13 allocations: 1.703 KiB)
Eigen{Float64, Float64, Matrix{Float64}, Vector{Float64}}
values:
2-element Vector{Float64}:
 1.585786437626905
 4.414213562373095
vectors:
2×2 Matrix{Float64}:
 -0.92388   0.382683
  0.382683  0.92388

you can see it's much faster. Indeed, @time is not an accurate way to measure such short times; a better option is to use the BenchmarkTools package, which provides @btime (benchmark time) and yields an accurate result:

julia> using BenchmarkTools

julia> @btime eigen([2 1; 1 4])
  826.000 ns (13 allocations: 1.70 KiB)
Eigen{Float64, Float64, Matrix{Float64}, Vector{Float64}}
values:
2-element Vector{Float64}:
 1.585786437626905
 4.414213562373095
vectors:
2×2 Matrix{Float64}:
 -0.92388   0.382683
  0.382683  0.92388

Thus, less than a microsecond is required. You might be skeptical--if you try this, you'll note it takes far longer than a microsecond to run @btime--but the explanation is that @btime gets its accurate measurements by running the code repeatedly and computing the average.

One consequence of the compile-time overhead is that it is not recommended to run performance-sensitive calculations with scripts from the command line: each run will be compiled freshly, and so you pay the cost of compilation each time you run it. Instead, start a Julia session and run the command repeatedly from within the same session, so that you only pay the compile-time cost once. You can use tools like Revise.jl or inline evaluation with VSCode or Juno to continue to evolve your code while keeping your session running.

[ChrisRackauckas]

I kind of wanted it to be a bit heavy, making it explicit in the manual that it's not up for debate one's compilation-inducing workflow is an appropriate way for people to use (and thus benchmark) quick codes that are compile-time dominated: I think the documentation should specifically say you should only do that if you absolutely don't care about performance and it's not recommended. I think it guides users in the right direction and gives a very explicit statement that if someone is benchmarking code like that, they are directly contradicting the best practices as laid out in the manual, not just some best practices that most people on Discourse etc. seem to follow.

Incorporating it into the @time section was something I considered, but I think it's slightly separate from just timing. It's of course useful in timing to remove compilation time from the process, but workflows that reduce the amount of compilation are useful and recommended for cases beyond timing. Even when you aren't timing, I think it's good to recommend a style that is Revise-based, open REPL, or PackageCompiler, since otherwise I think the user will not get as great of an experience. So it overlaps with the timing section but I don't think it's quite the same thing.

Also, making it all about timing makes it look like a trick to make timings look better: I've seen people write it off and not time twice in benchmarks because they believe that's not representative of how running code actually works. If the manual is explicit that the recommended workflows are these styles that do not require recompilation, then it immediately follows that the appropriate way to time is to time twice to not time compilation.

[IanButterworth]

Would it be possible for @time to report any compilation time, perhaps as a % when greater than zero, like gc reporting behaves?

Edit: Implemented in #37678

[timholy]

That would be useful, but I think what Chris is worried about is

$ time julia -e 'my_code.jl'

i.e., Linux-time rather than Julia-@time.

[timholy]

In that case, here are a few more specific comments. Even with those comments, it seems to approach its underlying message a bit indirectly. What about coming out and saying something along the lines of "this is not how Julia is intended to be used"? And then finish with your "If you must use Julia this way, and if the compile performance is an obstacle, use PackageCompiler.jl..."

I would also recommend using links to these tools rather than assuming people will find them from the name. Just omitting the .jl from a google search can lead to failure.

[timholy]

JIT has not yet been defined on this page

[timholy]

This sentence seems awkward; I think it's because the emphasis position of the sentence, the end, seems to veer off in a new direction.

[timholy]

In what context? If I have a computation that runs for 3 days that might be performance-sensitive, yet there's little reason not to launch it from the Linux prompt because compilation might take 20s.

[timholy]

it's more that it affects your measurement; in a sense your argument is that it doesn't matter (or rather that it is an orthogonal concept)

[timholy]

the first bits (Revise and the IDEs) are essentially ways to achieve "keeping REPL sessions alive," so I'd invert the order here and start with the goal and then mention tools that can help you achieve it.

"inadvisable and not a recommended" seems redundant

[timholy]

In what context is "performance necessary" for things that complete quickly? Maybe just say "when benchmarking"? Or if you have other cases in mind too, perhaps spell those out?

[timholy]

Maybe save this sentence for the end: it's forward-looking (it tells you how to solve the problem), but then you go back and tell people not to do the bad thing. Preserving the temporal order might help clarify your meaning.

[timholy]

I'd put all your recommendations for fixing the issue in a block.

[KristofferC]

I like the idea but this feels a bit heavy for what it's trying to convey, while also a bit disconnected from the rest of the performance tips page

This was my initial reaction as well. Compilation time is already discussed in earlier sections so I don't really see what this provides. And, to me, this comes of a bit like a "you're holding it wrong" type of comment that people will start linking as soon as someone complains about compilation time. The tone feels more suitable for a blog or a medium like that.

[DilumAluthge]

The tone feels more suitable for a blog or a medium like that.

If I understand correctly, @ChrisRackauckas specifically wants this this to be in the manual so that when someone is running e.g. some kind of benchmark competition between languages, we can point them to the manual as an official/authoritative way of saying "your benchmarking claim is not correct."

[matthieugomez]

I don’t think it is inherently wrong to complain about compilation time in certain packages (as long as users clearly identify it). For instance, packages with over parametrized types can lead to large compilation times that do not disappear over time — I think this is why DataFrames decided not to be strongly typed in the end.

[ViralBShah]

One of the things we are really missing, is a description of Julia's compilation stages in the manual. Something we can really point users to - a model that they can hold in the mind around what happens during precompilation, what happens during compilation, what the JIT does, what is included in a system image, and then the differences between a first and second run.

I know this is not the point of this PR, but I think having this would greatly help a lot of users, and then the point about how to benchmark and so on can be included as well.

[DilumAluthge]

@ChrisRackauckas and @timholy What is the status of this PR, and what needs to be done?

It would be great to get something like this into the manual.

[DilumAluthge]

Also @ChrisRackauckas could you rebase this on the latest master?

[timholy]

(I removed my name from requested reviewers because I don't think it's changed since this review.)

[stillyslalom]

My take on the wording for this PR, focused more towards new user experience - I don't think plotting is 'performance-sensitive' in most cases, but plot latency is a bigger pain point than microbenchmarking.

Recoup the cost of compilation overhead by keeping a Julia session alive

When using simple functionality from large packages (including most plotting and data-processing libraries), it can take several seconds to load and compile functions from a given package, even though subsequent use might take fractions of a second. This is known as the "time to first plot", and it arises from the flexibility (and attendant complexity) of dynamic multiple dispatch, which prevents Julia from simply caching the results of compilation for re-use between sessions.

Several options are available to help avoid paying the cost of compilation more often than necessary. The easiest is to keep a Julia session open, typically using Revise.jl to automatically replace modified function definitions whenever you save your source files. If you need to repeatedly call a Julia script from the command line, try DaemonMode.jl, which uses a client/server model with a Julia process running in the background. For deployed applications, as well as 'static' workflows involving a fixed set of packages, you can use PackageCompiler.jl.

[vtjnash]

I don't think this is particularly necessary or productive to include in the performance tips. In an ideal world, people would benchmark what matters. But I don't think we can beneficially tell people they are wrong for measuring something else, so it remains up to readers to decide if a particular benchmark report is sensible.