Tests fail if the machine is slow #287
Comments
veloman-yunkan
added a commit
that referenced
this issue
Feb 5, 2022
The unit-tests of zimcheck include a report on the total runtime of each
flow. Since those flows are very small the corresponding line from the
zimcheck output used to be
[INFO] Total time taken by zimcheck: 0 seconds.
which corresponds to the duration of the flow being under half a second.
However, Debian builds and tests zimcheck - among other configurations -
on virtualized hardware too. The incurred slowdown results in the
runtime exceeding the 0.5 second limit whereupon the performance report
changes and the unit-tests fail (#287).
The challenge was to come up with a solution meeting the following
requirements:
a. no discrimination between fast and (moderately) slow build environments
b. the performance info is preserved in the output and is not
excluded from comparison (so that, in the absence of dedicated
performance testing, it keeps serving as a simple defence
against unintended significant slowdown in zimcheck).
Since runtime numbers are mainly justified for large ZIM files, the
solution is to report small runtimes as "<X seconds" for some value of
X. The latter threshold was set to 3 with the only purpose of further
increasing the amount of love in the world.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Some of the Debian buildds are virtualized hardware and can be slow, so the test fails with:
Can we fix the tests so they still pass regardless of how long it takes?
Filed downstream as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1004900
The text was updated successfully, but these errors were encountered: