new cmd: nim r main [args...] to compile & run, saving binary under $nimcache/main

[timotheecour]

(updated description)

new cmd: nim r main [args...] now outputs binary under $nimcache/main and runs $nimcache/main [args...]

examples

nim r compiler/nim.nim --help # only compiled the first time
echo 'import os; echo getCurrentCompilerExe()' | nim r - # this works too
nim r compiler/nim.nim --fullhelp # no recompilation
nim r --nimcache:/tmp main # binary saved to /tmp/main

benefits

• don't clobber source dirs with binaries; they're hard to gitignore on posix (no extension) without jumping through some hoops, and in fact I don't know anything that works without breaking other tooling, as I explained here; https://stackoverflow.com/questions/61381438/how-to-gitignore-extension-less-files-without-breaking-a-global-gitignore
• (once --mode:js is added in future PRs) don't clobber source dirs with generated js files (hard to gitignore because their filename looks exactly like a regular source js file)
• adhers more to out-of-source builds principle

This can become the new preferred way to write tests in nims/nimble files, avoiding the problem of clobbering your repo with binaries (or fighting gitignore on posix)

related issues

• related to RFC: nim c output path RFCs#29 and nimble test, nimble build etc should build all artifacts under ./build nimble#787

same as in some other languages

• D: dmd -run main.d => outputs in some temp dir
• D(rdmd): rdmd main.d => outputs in some temp dir hashed by all relevant options
• go: ditto

package main
import ("fmt"; "os")
func main() {
    ex, err := os.Executable()
    if err != nil { panic(err) }
    fmt.Println(ex)
}

• C via tcc: tcc -run test.c
  => doesn't generate a binary (and perhaps does all in memory but that's besides the point)
• ditto with scripting languages (python, swift, node js, octave, matlab etc) which don't generate clobbering artifacts when you run python main.py
• lua via lua test.lua

future work

• nim r --mode:cpp main (EDIT: this was implemented by fix some issues with --backend #14363, via --backend)
• hash-based-caching (see --usenimcache:usehash timotheecour/Nim#199) => --usenimcache (implied by nim r main) now caches some compile options to avoid recompiling when project was previously compiled with such options. #17829

links

• what's missing in nim r? maybe recommend as an alternative for nim >= 1.4 in the readme? PMunch/nimcr#14

[Varriount]

I'm not sure I like this. When I compile something with the -r flag, I don't expect it to be placed in a completely different directory. The fact that *nix tools don't account for the name style of binaries is not insurmountable, just annoying.

[Clyybber]

I know this is kinda WIP, but breakage that requires workarounds like this is really bad: https://github.com/jangko/msgpack4nim/pull/47/files#diff-742733ae27de3fec3ce193ca005475b5R26

[timotheecour]

I know this is kinda WIP, but breakage that requires workarounds like this is really bad: jangko/msgpack4nim/pull/47/files#diff-742733ae27de3fec3ce193ca005475b5R26

I know, and I've now fixed it in a better way, see jangko/msgpack4nim#48
note that this was the only breakage among all 82 packages we're currently testing in nim CI.

note that I've noticed that --outdir:'$projectdir' is not supported on windows; so I've used explicitly --outdir:somedir in msgpack4nim; if needed to unblock this PR I can add a fix for --outdir:'$projectdir' on windows

When I compile something with the -r flag, I don't expect it to be placed in a completely different directory

you can use --outdir:somedir to place it where you want (see also above note). If needed I could also add a --oldoutdirlogic which would revert back the behavior prior to this PR (and would be safe to put in your ~/.config/nim/config.nims as a global flag) but hopefully that's not needed.

One of the benefits of this PR is we now don't need to worry about adding some nontrivial platform specific logic (especially on posix with empty extensions; note that some source files may have no extensions) in .gitignore just to support binaries / generated js file generated by running test commands like:

nim c -r test
nim js -r test

which affects pretty much every nimble package.

it also prevents clobbering your sources with temporary files (exe's + generated js files)

[Clyybber]

note that I've noticed that --outdir:'$projectdir' is not supported on windows; so I've used explicitly --outdir:somedir in msgpack4nim; if needed to unblock this PR I can add a fix for --outdir:'$projectdir' on windows

Yeah, please do.

[timotheecour]

Yeah, please do.

=> #13407 /cc @Clyybber

[Araq]

I don't like this code breakage. Instead currently nim foo.nim (Note: Without the c) is an invalid command. It could instead mean "nim c -r --output:$nimcache" (What your PR does). Or maybe nim x foo.nim.

[timotheecour]

that's a good idea; I'll do the following:

nim r foo.nim # means: nim c -r --outDir:$nimcache foo.nim

where r reminds of -r

future PR's (maybe)

• --cache: caches instead of overwriting, using a dir inferred from options. (EDIT: tracked here --usenimcache:usehash timotheecour/Nim#199)
  It's orthogonal (but complementary) to r and can be used with or without it, with hopefully obvious semantics

nim r --cache -d:foo --gc:arc foo.nim # means: nim c -r --outDir:$nimcache/mangledoptions/ foo.nim

where mangledoptions is a hash of cmdline params, reusing similar logic as the one that decides whether to recompile (betterRun), so that you can cache multiple invocations of nim without clobbering previous ones. That's basically the rdmd main.d logic, which allows running multiple programs (or options) without needing to recompile each time you switch (only once per unique invocation).

• --mode: this could be useful too in case the default mode (c) is not desirable:

nim r --mode:js foo.nim # means: nim js -r --outDir:$nimcache foo.nim

(EDIT: done in #14278)

[Araq]

Please finish this PR. nim r foo.nim is good.

[alaviss]

• --mode: this could be useful too in case the default mode (c) is not desirable

Better yet, decouple the concept of target from commands in the compiler, so we can finally have nimsuggest working on js projects.

[timotheecour]

Please finish this PR. nim r foo.nim is good.

@Araq PTAL. this now works, and is now the preferred way to write tests in nims/nimble files, avoiding the problem of clobbering your repo with binaries (or fighting gitignore on posix)

  nim r compiler/nim.nim --help # only compiled the first time
  echo 'import os; echo getCurrentCompilerExe()' | nim r - # this works too
  nim r compiler/nim.nim --fullhelp # no recompilation
  nim r --nimcache:/tmp main # binary saved to /tmp/main

nim r --mode:cpp main and hash-based-caching is deferred to future PR's

[alaviss]

Can you update the PR description? This is no longer a breaking change to nim c -r. Would help when someone go back and read this PR.

[timotheecour]

@alaviss

Can you update the PR description? This is no longer a breaking change to nim c -r. Would help when someone go back and read this PR.

done