Stress testing of the engine

[egavrin]

Do you know any fuzzers or stress testing suites that can useful for us?

• jsfunfuzz - https://www.squarefree.com/2007/08/02/introducing-jsfunfuzz/
• afl-fuzz for javascript - https://github.com/tunz/afl-fuzz-js

[akosthekiss]

The University of Szeged has a proprietary fuzz testing framework with no imminent plans for open sourcing but is happy to share any issues it discovers in Jerry. (Actually, several previously reported issues were discovered with its help.)

[renatahodovan]

The "problem" with jsfunfuzz is that it generates the test cases on the fly which means that the whole generator script needs to be loaded by the engine every time. Hence jerry would need to load ~5500 LOC to generate a single test case but unfortunately it runs of the registers quickly.

I've also tried the recently popular afl fuzzer against jerry however it constantly crashes on the longjmp bug (#341) and it's unusable this way.

[ruben-ayrapetyan]

@renatahodovan, maybe we can generate tests by running jsfunfuzz separately, and then use the generated tests for checking.

I've also tried the recently popular afl fuzzer against jerry however it constantly crashes on the longjmp bug (#341) and it's unusable this way.

Could you, please, check #390?

[ruben-ayrapetyan]

#341 is fixed.

[egavrin]

@renatahodovan could you please share how to run afl-fuzz for javascript?

[renatahodovan]

@egavrin sure, however I'm also still working on how to exploit all of its features.

A customized version of afl-fuzz is available from here: https://github.com/tunz/afl-fuzz-js
You can run it two different ways: in the first case you can use your own jerry build, choose a set of initial test cases that will be mutated and a predefined dictionary to guide the mutation and go ahead. From the manual, starting a fuzz session looks something like this:

/path/to/afl-fuzz -i [input directory] -o [output directory] -x [dictionary directory] -m 8G ./jerry @@

I've ran this for days but it hasn't found much (except the longjump crashes).

However, there is second coverage driven option (what is said much more efficient). The problem is that to use this option the target must be instrumented with the afl-fuzzer's own wrapped gcc/g++. Unfortunately, replacing jerry's default compiler is not seems too easy. I've achieved to build everything with afl-g++, but it still fails on linking. This is what I'm experimenting with right now :)

[akosthekiss]

There is an experimental branch in my repo for overriding the default compiler (https://github.com/akiss77/jerryscript/commits/compiler-override) but it's not ready for PR yet. Feel free to (ab)use it directly from there though. (However, even that experiences the linker issue.)