Understanding inconsistent coverage reports

[renatahodovan]

I'm attempting to comprehend why the project, which I aim to enhance, yields quite inconsistent coverage results. Upon examining the coverage reports, it seems that in case of low coverage, one or two of the three possible targets are called only a few times (less than the number of seeds in the initial corpus; see example here: although the corpus contains 21 elements but LLVMFuzzerTestOneInput was executed only 12x). I suspect that some flaky timeout or OOM occurs during corpus loading, causing the fuzz target to terminate prematurely. Unfortunately, I cannot validate this locally with the helper script. Therefore, I'm interested in whether it's feasible to access the fuzzer logs associated with the public coverage reports, or at least one of the logs accompanying the low coverage report. Alternatively, if someone could provide guidance on how to reproduce the CI setup using the infra/helper.py script (or run the CI itself locally), including timeout, RSS, max_total_time, environment variables, etc., that would be greatly appreciated.

[DavidKorczynski]

I could imagine this is likely due to some statefulness in the target? See here for an issue that looks into a similar issue: #9928

In short, the harness should ideally execute the same set of code independently of what order the corpus is run against the target, however, I would suspect in this case there is some statefulness that means the order impacts what is being executed. This is kind of similar to your suspicion regarding timeouts or OOMs.

This page may help re coverage logs: https://oss-fuzz-build-logs.storage.googleapis.com/index.html#quickjs

[renatahodovan]

@DavidKorczynski Thanks for your reply! There was a statefulness issue in the targets indeed which I fixed just yesterday. I thought that it was responsible only for the irreproducible issues and not for the coverage inconsistencies. However, if this is the case, then the coverage should stabilise as soon as the new code starts running in the next days.

This page may help re coverage logs: https://oss-fuzz-build-logs.storage.googleapis.com/index.html#quickjs

I knew this build log page, but it doesn't contain information about the execution parameters of the targets. Could you validate that the coverage results are generated after 10 minutes of fuzzing with 25seconds of timeout (I saw similar constants somewhere, but I wasn't sure that these are actually used to generate the corpus for coverage measurement)?

[DavidKorczynski]

Yeah, if there is statefulness then this will impact coverage collection, I'm quite that's the root cause of this issue. It will also have affected, e.g. corpus pruning which means that the corpus may have jumped a bit in size sporadically. Assuming statefulness have been resolved then I think you should start seeing more stable/reliable patterns on the coverage graph.

Regarding coverage collection, this is the specific line used for running the actual coverage extraction:

oss-fuzz/infra/base-images/base-runner/coverage

Line 101 in 889d0c5

timeout $TIMEOUT $OUT/$target $args &> $LOGS_DIR/$target.log

Timeout per target is set to 100sec:

oss-fuzz/infra/base-images/base-runner/coverage

Line 98 in 889d0c5

local args="-merge=1 -timeout=100 $corpus_dummy $corpus_real"

[maflcko]

Timeout per target is set to 100sec

There is an open timeout issue in at least one of the fuzz targets: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22680&sort=-id&q=proj%3Aquickjs&can=2, so that the timeout is also causing the coverage issue seems plausible.

[renatahodovan]

@maflcko Thanks, I'll try to look into it.