Configuration via env variables

[kornelski]

I need to git bisect a performance regression, and too late after the fact I've realized that the default of benchmark with 100 iterations takes way too long to finish the bisect in a reasonable timeframe. The difference I'm measuring will be measurable with just a handful of iterations, so I would be beneficial if I could lower it for the bisect operation. The problem is that change of Criterion's configuration requires significant source code changes. This is especially inconvenient when measuring past commits that can't be simply modified without some git wrangling.

My second use-case is setting a baseline without running into #193. In bigger workspaces it's hard to track down all the places that break the custom command-line arguments. Setting extra criterion options via env vars would easily bypass the picky test runner.

[kornelski]

PLEASE offer an alternative to the command-line arguments. They are fundamentally broken and unfixable.

For example, https://github.com/gfx-rs/wgpu/blob/trunk/benches/Cargo.toml does everything right, and still doesn't work:

error: unexpected argument found
error: bench failed, to rerun pass `--bench root`

[rben01]

I would also like the ability to specify the options here via command line args or env variables. In my case I'd simply like to globally configure every benchmark without needing to add config = Criterion::default().significance_level(0.1).sample_size(500); to each and every use of criterion_group!. (Of course, a config provided in source would take precedence over command line args or env vars.)

@kornelski regarding criterion-specific command line arguments breaking things, I believe you must set harness = false for each and every benchmark in benches/ — that's one [[bench]] section per Rust file in benches/, with name = filename\nharness = false.

[kornelski]

In large/complex workspaces it's frustratingly difficult to find all the places that need the harness setting, and it's even more cumbersome when bisecting and past commits didn't have the required harness setting.

So I know a solution exists, but to me it's not usable. I'd rather avoid using command line args entirely than fight with the Cargo/rustc harness.

[rben01]

100% agree, was just pointing out the (annoying but guaranteed to work) fix.

[aidanhs]

For other readers - after struggling with the harness argument handling I've ended up just putting my benchmarks in a cargo example. This also lets me do some global initialisation.

fn main() {
    criterion_main!(benches);

    env_logger::init();
    main() // calls the criterion main
}