ARC: passed tests marked as failed when running sanitycheck on nsim_* platforms

[evgeniy-paltsev]

Describe the bug

The 531fe89 commit of #28372 causes regression for ARC.
When we running sanitycheck on nsim_* platforms we get some exception from sanitycheck code and mark test as FAILED however it successfully passed.

Here is an example on exception:

Exception in thread Thread-15603:
Traceback (most recent call last):
  File "/home/paltsev/CodeData/zephyr/zephyr-arc-64/zephyr/scripts/sanity_chk/sanitylib.py", line 496, in _output_handler
    proc.wait(0.1)
  File "/usr/lib64/python3.7/subprocess.py", line 1019, in wait
    return self._wait(timeout=timeout)
  File "/usr/lib64/python3.7/subprocess.py", line 1645, in _wait
    raise TimeoutExpired(self.args, timeout)
subprocess.TimeoutExpired: Command '['make', 'run']' timed out after 0.1 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/paltsev/CodeData/zephyr/zephyr-arc-64/zephyr/scripts/sanity_chk/sanitylib.py", line 498, in _output_handler
    self.terminate(proc)
  File "/home/paltsev/CodeData/zephyr/zephyr-arc-64/zephyr/scripts/sanity_chk/sanitylib.py", line 456, in terminate
    os.kill(child.pid, signal.SIGTERM)
ProcessLookupError: [Errno 3] No such process

ERROR   - nsim_em                   tests/kernel/mutex/sys_mutex/system.mutex          FAILED: Failed
ERROR   - see: /home/paltsev/CodeData/zephyr/zephyr-arc-64/zephyr/sanity-out/nsim_em/tests/kernel/mutex/sys_mutex/system.mutex/handler.log

In the tests/kernel/mutex/sys_mutex/system.mutex/handler.log everything looks OK:

 [-snip-]
[100%] Built target zephyr_final
Scanning dependencies of target run
Console now belongs to UART, hit CRTL-] to return to simulator.
*** Booting Zephyr OS build zephyr-v2.4.0-1942-g531fe89e80f8  ***
Running test suite mutex_complex
===================================================================
START - test_mutex
START - Test kernel Mutex API
===================================================================
Done LOCKING!  Current priority = 5
Testing recursive locking
Recursive locking tests successful
 PASS - test_mutex
===================================================================
START - test_user_access
E: syscall check_sys_mutex_addr failed check: Memory region 0x80000c08 (size 4) write access denied
E: syscall check_sys_mutex_addr failed check: Memory region 0x80000c08 (size 4) write access denied
 PASS - test_user_access
===================================================================
START - test_supervisor_access
 PASS - test_supervisor_access
===================================================================
Test suite mutex_complex succeeded
===================================================================
PROJECT EXECUTION SUCCESSFUL

The tests fail randomly. Usually 2-9 tests fail for nsim_em for each sanitycheck run.

To Reproduce
Lunch sanitycheck for nsim_em with toolchain usage from Zephyr SDK.

./scripts/sanitycheck --platform nsim_em

Impact
The random failing of tests on nsim_* platforms block it usage in verification.

Environment (please complete the following information):

• OS: Linux
• Toolchain: Zephyr SDK 0.11.4 (same situation with 0.12.0-beta-1)
• nSIM: 2020.09 (same situation 2020.06)
• Commit: 531fe89 (or later)

[evgeniy-paltsev]

The fix for this is #30411, so no need in debugging.

[ruuddw]

I believe PR #30411 actually fixes this - however it needs an update/rebase for the sanitcheck->twister rename.
@wangnuannuan can you please take care of that?