Run the official ECMAScript (ECMA-262) conformance suite.

[LaszloLango]

We should be test engine on the original testsuite without any modification. Issues that have to be solved for this:

• RegExp engine Implement ECMA262 RegExp builtin object and RegExp engine #27
• Unicode support Unicode support #49
• Able to parse JS files from 70 Kbyte to 150 Kbyte
• Implement for-in loop Implement for-in statement #51
• Fix the for loop bug with function calls in the iteration part PARSE_ERROR in for loops #156

These are only the currently known issues, there could be more.

Testsuite

Test262

[egavrin]

@LaszloLango how can I make official test suite runner to print only short summary? Like that:

./tools/runners/run-test262.sh ./build/bin/release.linux/jerry
OK: 6573 (55.9357 %)
FAIL:   5178 (44.0643 %)

The following command generates full log for me:

tools/packaging/test262.py --command ../../build/bin/release.linux/jerry

[LaszloLango]

for example:
tools/packaging/test262.py --command ../../build/bin/release.linux/jerry --summary | grep -A3 '=== Summary ==='

[zherczeg]

Let me give a bit more insight about the problem. We need an independent testing suite to report the progress on Jerry. It is not difficult to achieve 102% OK on our testing suite, but that doesn't mean our engine is more than perfect. So we need something created by other people. Test262 is a widely accepted testing environment.

We are aware, that there is a run-test262.sh in Jerry. However, it only works with a specific version of test262 (it only reports 4% OK for a newer version), and it reports 45% OK for another engine, which otherwise passes all test262 tests. So this script might not be the best to track the progress. We could fix it, but I am not sure that is the right way. I would personally prefer to run the real test262 environment created by external people. I feel that would show our progress better.

[tilmannOSG]

Yes, being able to run the official test262 test driver script is a strong requirement. Otherwise there's no reliable way for third parties to assess the level of conformance of the engine.

[egavrin]

We could fix it, but I am not sure that is the right way.

@zherczeg I'm absolutely convinced that this is not necessary. Obviously, we have to run official test262.

[zherczeg]

After the fixes I tried to run the official test suite, but there is an out-of-memory error. The sta.js and testIntl.js is too much for Jerry. Could we extend the memory amount to a few megabytes?

[egavrin]

The main issue, right now is testIntl.js
Currently, the only changes for me is to make testIntl.js empty and remove test/suite/intl402/*.

These files are related to support of other standard I think it's OK for now. We plan to remove memory limit at Sept./Oct, since it's more important to improve performance and reduce memory consumption, both for Jerry and IoTjs.