Conclude phase (opposite of --prepare)

[gcflymoto]

In the same manner as the --prepare phase exists, the feature request is for a non-timed --conclude or --post phase that would run after each benchmarked command. There would each be 1 or N number of these.

In my use case, I cannot do the cleanup after the command executes W(armup)+N times. I also understand that running in this phase can pollute the cache, but in my case, the benchmarks can already run for hours.

The current workaround is to put the per command shutdown steps in each bechmarked command but that pollutes the benchmarking times.

Thanks again for an excellent utility

[shoffmeister]

My use case, right now, is benchmarking server startup performance - e.g. "time to successfully serve the first request".

In terms of phasing, I'd use

• prepare ==
• command == bash: start server, busy-loop with curl until HTTP 200
• conclude == bash: zap server out of existence (pkill or whatever), this will release the server port

After enough warmup, the file system cache will be hot enough so that the only thing being measured would be the (CPU) time it takes for the server to start (plus, of course, the time to generate the response)

"conclude" (or whatever the name would be) establishes complete symmetry in phasing:

setup
    prepare
    conclude  <-- the missing piece
cleanup

[shoffmeister]

Coming to think of it, I can actually make do with only the prepare phase:

• prepare == kill (any) running server
• command == launch server, ...
• cleanup == kill (any) running server

This is workable, but ... does not feel nice ... compared to

• command == launch server, ...
• conclude == kill running server

[gcflymoto]

It's the current workaround I am using as well, but it means both the prepare and cleanup phase need to have duplicated mechanics. Going through and understanding what is actually happening is already not-clear to new comers.

[sharkdp]

Thank you for the feature request. I can certainly see the use case for this and agree that it would lead to a better UX.

Before someone goes ahead an implements this: it might make sense to introduce some useful abstractions to avoid a lot of code duplication for all of those options (--setup, --prepare, --conclude, --cleanup).

Naming wise, I think --conclude is not bad. Are there any alternative proposals? To avoid too much confusion, would it make sense to introduce a new naming scheme? Something like

• --before-benchmark <cmd> (for --setup <cmd>)
• --before-run <cmd> (for --prepare <cmd>)
• --after-run <cmd> (for --conclude <cmd>)
• --after-benchmark <cmd> (for --cleanup <cmd>)

"conclude" (or whatever the name would be) establishes complete symmetry in phasing:

setup
    prepare
    conclude  <-- the missing piece
cleanup

Yes. See also: https://github.com/sharkdp/hyperfine#detailed-benchmark-flowchart

[gcflymoto]

I think --conclude is not bad. Are there any alternative proposals?

--wrapup, --shutdown, --finish are a couple more suggestions

To avoid too much confusion, would it make sense to introduce a new naming scheme

Are you suggesting both sets of names would be available? ie there would be aliases? If not is there any concern for breaking backward compatibility with the new names?

[shoffmeister]

I just took a very quick stab at (blindly) implementing --conclude, see master...shoffmeister:hyperfine:feature/add-conclusion

Those changes are plenty ugly, much duplication, so I now understand the need domain expert's ;) perspective of there being a desire to

introduce some useful abstractions to avoid a lot of code duplication

I might find some minutes to fix my mess.

With respect to naming:

• removing the existing command-line parameters breaks the (user) contract, hence that would not be an option I would consider
• adding a new command-line option needs to be in line with the current single-word philosophy for the sake of consistency; conclude seems to be as good as any other informal working title until the bike-shedding is done?
• introducing a before and after pattern sounds like a very good idea, as it would improve UX; my gut feeling here is that
  • this should happen in a separate PR / issue (it is orthogonal)
  • before/after would become the primary documented command-line options and the single-word variants would be demoted to aliases (because of UX)

[gcflymoto]

This sounds like an excellent plan