[tooling] cache compilation to avoid recompiling when possible

[timotheecour]

The proposal is to introduce a flag (say,-cache) to cache compilation results including all needed artifacts:

• binary
• c/c++ files (etc)
• object files
• json file with dependencies

This feature is available in D via rdmd: when running the same thing twice, assuming nothing that could impact compilation result changed, the compilation step is omitted the 2nd time, resulting in instantaneous re-runs (useful for scripts among other things)

implementation

• A1: when nim c -cache foo.nim foo bar gets run, nim creates a hash string hashid of all compiler flags (including cmd line ones and implicit ones from nim.cfg, config.nims, but excluding program arguments given after foo.nim, these are not cached so we can rerun with different such arguments)

• A2: it then checks whether dir = nimcache/$hashid exists:

• A21: if $dir exists, it looks at json=$dir/deps.json (currently this file is called nimcache/foo.json when compiling foo.nim, but under this proposal there would be 1 json for each $dir, so let's just call it deps.json)

• A211: if all dependencies mydep mentioned in $json are up to date (ie mydep still exists and mydep.modificationTime < $json.modificationTime), then nim skips compiling and runs the binary (which is stored as exe = $dir/exe) with arguments foo bar

• else (if they're not up to date), nim compiles foo.nim as usual, outputting everything under $dir

• A212: else ( if $dir doesn't exist), it creates it and proceeds as in A211

garbage collection

in A212 step, to avoid using too much disk space, it also keeps track in nimcache/cache.json of an LRU cache, and removes old $dir folders according when either the number of such $dir folders is greater than a threshold, or when the time of last use (updated in A21 step) is older than an offset from current time

related

I had actually opened an issue mentioning this, but it was closed as completed (other aspects were completed, but not this part), so I'm creating a more targeted issue just to track this

generate the binary in nimcache, caching the resulting binary

• [TODO] could nimcache be generated out-of-source (in /tmp) like rdmd does in D? [TODO] could nimcache be generated out-of-source (in /tmp) like rdmd does in D? Nim#7402

• Create binary file in temporary folder. Nim#8910 is also related but deals with nim c -r foo.nim generating results under a nimcache folder, with no mention of caching binary output to avoid recompilation

links

• see also https://github.com/marler8997/rund which was litterally just released as a faster rdmd /cc @marler8997

[Araq]

Unclear to me what this is about. Things are already cached and recompiled on demand.

[timotheecour]

@Araq

Things are already cached and recompiled on demand.

only C files, not nim files, which are being re-parsed/re-compiled regardless they haven't changed:

main.nim

import os
static: echo "compiling"
echo commandLineParams()

nim c -r t0141.nim hello world
compiling
@["hello", "world"]

nim c -r t0141.nim hello world
compiling
@["hello", "world"]

this feature would enable using 0-install binaries, recompiled on demand when anything changes, otherwise using the cached binary; the overhead over running a binary directly would be minimal and equal to running stat on each (pre-computed) dependency of main.nim (we can customize excluding stdlib, if needed, to avoid running stat on stdlib files, if needed, but even in this case shouldn't bring a significant slowdown)

the approach would use same technique as #510 but with ``nim genDepend/--genDeps` to get the dependencies

[Araq]

Do you really think that I don't know that? We're working on --incremental:on for a reason.

[Araq]

Duplicate.

[timotheecour]

related: nim-lang/Nim#11709