Compatiblity mode: logs on libfuzzer and Seeds

[Alami-Amine]

• When I run FuzzTests using FUZZTEST_COMPATIBILITY_MODE, and my FuzzTests have seeds added through FuzzTest Macro with a something like return Arbitrary<vector<uint8_t>>().WithSeeds({ data });

1. in the Log it says A corpus is not provided, starting from an empty corpus, I assume this line should not be printed. I am sure that the seeds are working since I am going doing the valid path directly at the beginning, which would otherwise not be possible.

FUZZTEST_PRNG_SEED=t17e2bumtZy3CZf4snPiSGPXcgjJPryS9QIGA5Dn474
INFO: found LLVMFuzzerCustomMutator (0x61abd85c43b0). Disabling -len_control by default.


WARNING: unrecognized flag '-fuzz=FuzzCASE_PW.HandleSigma3b'; use -help=1 to list all flags

INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 2670306577
INFO: Loaded 1 modules   (17434 inline 8-bit counters): 17434 [0x61abd877f6d8, 0x61abd8783af2), 
INFO: Loaded 1 PC tables (17434 PCs): 17434 [0x61abd8783af8,0x61abd87c7c98), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2	INITED cov: 5 ft: 6 corp: 1/1b exec/s: 0 rss: 46Mb
fuzzed HandleSigma3b: src/lib/core/TLVReader.cpp:894: Error 0x00000022
Valid HandleSigma3b: src/protocols/secure_channel/CASESession.cpp:2100: Success

2. At the end of the log, I am not getting all the stats like Edges covered and Total edges

=================================================================
=== Fuzzing stats

Elapsed time: 24.130049416s
Total runs: 17906
Edges covered: 0
Total edges: 0
Corpus size: 0
Max stack used: 0

[xinhaoyuan]

Unfortunately the compatibility mode was implemented with best-effort to provide a way to run fuzz tests with libfuzzer or other engines. It does not guarantee to support all FuzzTest features, specifically the seeds in FUZZ_TEST().WithSeeds(), but that does not affect domain seeds FUZZ_TEST().WithDomains(SomeDomain().WithSeeds(...)) - this would still work.

I notice that there is a warning unrecognized flag '-fuzz=FuzzCASE_PW.HandleSigma3b' - you probably want to use --fuzz= instead.

At the end of the log, I am not getting all the stats

Unfortunately that is expected: in the compatibility mode the stats from the other engine is not propagated to the FuzzTest runtime.

[Alami-Amine]

@xinhaoyuan Thanks for clarifiying the difference between Domain seeds and FUZZ_TEST Macro seeds.

• the warning was just a typo on my end, althought the correct fuzztest was running.

---

I have two more questions related to ReadFilesFromDirectory():

1. In the Doc, it says that we can use .WithSeeds(fuzztest::ReadFilesFromDirectory() directly. However, this seems to be not (no longer?) the case, since ReadFilesFromDirectory returns a std::vector<std::tuple<std::string>> while WithSeeds() would need a std::vectorstd::string for this case (I got it work with a few lines of code). I guess the docs needs to be fixed?

2. Could I make a PR to create an overload of ReadFilesFromDirectory() that takes a string view to a regex pattern as a second argument ?

[Alami-Amine]

• FYI: I have created the PR here Adding an overload of ReadFilesFromDirectory that takes in a filename… #1520