Benchmarking for Qiskit
Benchpress itself requires no installation. However running it requires the tools in requirements.txt. In addition, running each of the frameworks has its own dependencies in the corresponding *-requirements.txt file
To run the benchmarks in the default configuration from inside the environment in which you want to perform the tests run:
python -m pytest --benchmark-min-rounds=1 benchpress/*_gymwhere * is one of the frameworks that you want to test, and which matches the environment you are in. Here --benchmark-min-rounds=1sets the minimum number of repeated trials to 1, which will save a great deal of time
To run the benchmarks and save to JSON one can do:
python -m pytest --benchmark-min-rounds=1 --benchmark-save=SAVED_NAME benchpress/*_gymwhich will save the file to the CWD in the .benchmarks folder
Further details on using pytest-benchmark can be found here: https://pytest-benchmark.readthedocs.io/en/latest/usage.html
Benchmarking the amount of memory a test uses can be very costly in terms of time and memory. Here we use the pytest-memray plugin. Calling the memory bechmark looks like:
python -m pytest --memray --trace-python-allocators --native --most-allocations=100 --benchmark-disable benchpress/*_gymHere --memray turns on the memory profiler, --trace-python-allocators tracks all the memoryu allocations from Python, --native track C/C++/Rust memory, --most-allocations=N shows only the top N tests in terms of memory consuption, and finally --benchmark-disable turns off the timing benchmarks.
The pytest-memray plugin will sometimes raise on building the histrogram included in the report by default. Currently the only way around this error, which does not affect the tests, is to manually comment out L322 and L323 from the plugin.py file:
#histogram_txt = cli_hist(sizes, bins=min(len(sizes), N_HISTOGRAM_BINS))
#writeln(f"\t 📊 Histogram of allocation sizes: |{histogram_txt}|")Benchpress makes use of files from the following open-source packages under terms of their licenses. License files are included in the corresponding directories.