Performance regression after #8447

[jchavarri]

While testing most recent commits from main I noticed that noop builds were sustantially slower in the last weeks.

After dissecting for the past few months, I could pin down to commit 9105a43, from #8447, cc @rgrinberg @Alizter as PR author and reviewer.

The regression makes the noop build in Ahrefs case almost 2x slower:

# using commit 9105a43b719cc66aefd7c7f84816bd1a77f236c7
$ time ../dune/_build/default/bin/main.exe build @fe.all
                                             
real    0m43.125s
user    0m41.054s
sys     0m2.092s

# using commit 9c41108359ac93d369a6e78da88ffbde8b1f8ca9
$ time ../dune/_build/default/bin/main.exe build @fe.all
                                             
real    0m22.828s
user    0m21.099s
sys     0m1.760s

All other things in the measurements above are equal. I am building the dune exe using dune build bin/main.exe and calling the exe directly as shown in the time commands above.

I will investigate more tomorrow to see if I can identify which particular rules are slowing down, in case there are any other info I can provide, please lmk.

[jchavarri]

Here are some perf measurements, obtained with following commands:

perf record -F 1000 --call-graph dwarf ../dune/_build/default/bin/main.exe build @fe.all
perf script -F +pid > xxx.perf

• dune_9c41108359ac93d369a6e78da88ffbde8b1f8ca9_fe_all.perf.zip
• dune_9105a43b719cc66aefd7c7f84816bd1a77f236c7_fe_all.perf.zip

And a few screenshots, there are quite some differences between both, they seem to be concentrated in the rule loading stage:

9c41108	9105a43

[rgrinberg]

Simplest thing you could do to mitigate this, is to speed up the comparison of dune file ast's by including a checksum inside the dune file data structure. See Dune_engine.Dep for an example of how we do this.

[nojb]

@rgrinberg the slowdown seems pretty consequential (100% slower in no-op builds); is the reason for the slowdown understood? was a slowdown expected as a consequence of #8447?

[rgrinberg]

Somewhat of a slowdown as expected. The reason is understood. #8447 relies a lot more on the comparisons of dune files which is slow. We already rely on this comparison elsewhere so I thought it wouldn't make much of a difference.

Tbh, it's probably not a huge issue anyway. Incremental build performance for workspaces that large is pretty hopeless without the watch mode. I'm more interested in improving the watch mode to make it usable for more people, but of course I'll accept whatever optimizations people for 0 builds.

[nojb]

For what is worth, I tested LexiFi's codebase: a noop build went from 2.8s (with Dune 3.8) to 3.6s (with a version not very far from the current main).

[jchavarri]

Fixed in #9769.

[Khady]

Tbh, it's probably not a huge issue anyway. Incremental build performance for workspaces that large is pretty hopeless without the watch mode. I'm more interested in improving the watch mode to make it usable for more people, but of course I'll accept whatever optimizations people for 0 builds.

It is a huge issue. Watch mode just doesn’t cut it for now. There are multiple things that dune do not handle and require staged compilation for example. And the lock just blocks too many operations that require to stop the watch mode. Such a slow down also makes watch mode much more troublesome to use because it takes longer to start.

[rgrinberg]

All of those facts point to improving the watch mode. We've fixed the regression but a 0 build is still 20 seconds and optimizing it beyond that would require a lot of work. On the other hand, improving the watch mode is fairly straightforward.