OME Files Benchmark

The current repository contains the set of benchmark scripts used to test the performance of the OME Files library.

Available benchmarks

A few different benchmarks are available via this repository. Each benchmark contains its own set of tests and metrics. The following section describes how to build and execute the benchmark. In addition, each benchmark might require a set of predefined datasets.

Refer to the following pages for more details about the scope, datasets, tests, and results of:

• the metadata and pixeldata benchmark,
• the tiling benchmark.

Building and executing the benchmark scripts

See the OME Files C++ and Bio-Formats building instructions.

Windows

The Windows build requirements are Cmake, Maven, Visual Studio and a local version of the standalone OME Files bundle matching the Visual Studio version.

For running our builds, we used the Continuous Integration software Jenkins to trigger the Windows benchmark builds. A single script executing the building and execution steps is available under jenkins_build.bat.

To build the OME Files performance scripts manually, within a build directory, execute the following cmake command:

$ cmake -G "Ninja" -DCMAKE_VERBOSE_MAKEFILE:BOOL=%verbose%
  -DCMAKE_INSTALL_PREFIX:PATH=%installdir% -DCMAKE_BUILD_TYPE=%build_type%
  -DCMAKE_PREFIX_PATH=%OME_FILES_BUNDLE%
  -DCMAKE_PROGRAM_PATH=%OME_FILES_BUNDLE%\bin
  -DCMAKE_LIBRARY_PATH=%OME_FILES_BUNDLE%\lib 
  -DBOOST_ROOT=%OME_FILES_BUNDLE% %sourcedir% 
$ cmake --build .
$ cmake --build . --target install

The Bio-Formats performance script can be built within the source directory using Maven:

$ cd source
$ call mvn clean install

Linux

The Linux benchmark was performed on Ubuntu 16.04. To ease the distribution and reproducibility of the suite, the benchmark environment is built using Docker via a Dockerfile including the software prerequisites and the benchmark code.

To retrieve the Docker image, run::

$ docker pull openmicroscopy/ome-files-performance:latest

For testing, you can pull the Docker image built daily including all the Pull Requests:

$ docker pull snoopycrimecop/ome-files-performance:master_merge_trigger

To build the Docker image locally, run:

$ docker build -t ome-files-performance .

Create a unique benchmark directory which will be mounted as the /data volume when running the Docker container, e.g.:

$ mkdir /temporary/benchmark-DD-MM-YYYY

where DD-MM-YYYY would be the date of execution of the benchmark.

This structure of this benchmark directory will be the following:

• if the benchmark expects specific input datasets as specified in the ome-files-performance README.md, copy this data under the root folder of the benchmark directory e.g. /temporary/benchmark-DD-MM-YYYY/tubhiswt-4D/,
• the results generated by the benchmark will be stored under a subfolder called results,
• the temporary files generated by the benchmark will be stored under a subfolder called :file:out.

The default docker run command will execute the benchmarks defined by the scripts/run_benchmarking script without argument:

$ docker run --rm -it -v /temporary/benchmark-DD-MM-YYYY:/data openmicroscopy/ome-files-performance:latest

To execute a particular benchmark, start a shell in the Docker container using docker run --entrypoint, e.g.:

$ docker run --rm -it --entrypoint=bash -v /temporary/benchmark-DD-MM-YYYY:/data snoopycrimecop/ome-files-performance:master_merge_trigger

Then from the shell, execute the desired benchmark::

root@084bb88d5a62:/git/ome-files-performance$ ./scripts/run_benchmarking metadata

References

• OME Files documentation
• Bio-Formats documentation