We have other processes running on the dedicated machine and they might interfere with the benchmarks' timings.

We can work around that by giving the bot's benchmarks a higher priority, e.g.

sudo ionice -c 1 -n 0 sudo nice -n20 sudo -u benchbot cargo ...

Aside from ionice and nice, I've also heard about chrt but never used it myself.

I'm by no means an expert in setting process' priorities so I'd advise for the implementer to also consider other Unix tools aside from the ones listed above.