Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regression tests on macOS sometimes randomly fail #702

Open
s3rvac opened this issue Feb 8, 2020 · 0 comments
Open

Regression tests on macOS sometimes randomly fail #702

s3rvac opened this issue Feb 8, 2020 · 0 comments

Comments

@s3rvac
Copy link
Member

s3rvac commented Feb 8, 2020

This macOS build has succeeded. The one after that, which is for a commit that only changed the changelog file, failed because of the following test:

bugs.selective-decompilation-strange-behaviour.TestDecompileSelect0x0DecodeAll (main.ex --sel[..])  [ FAIL ]  (1.15s)
F
======================================================================
FAIL: test_has_no_functions (bugs.selective-decompilation-strange-behaviour.TestDecompileSelect0x0DecodeAll)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "retdec-regression-tests-framework/regression_tests/tools/decompiler_test.py", line 65, in setUp
    self.assertEqual(self.decompiler.return_code, 0, msg=msg)
AssertionError: 1 != 0 : decompiler failed; output:
...
Running phase: Simple types recovery optimization ( 0.65s )
Running phase: Stack pointer operations optimization ( 0.65s )
Running phase: Instruction idioms optimization ( 0.65s )
Running phase: LLVM ( 0.65s )
Running phase: LLVM instruction optimization ( 0.65s )
Running phase: Instruction idioms optimization ( 0.65s )
Running phase: Phi removal ( 0.65s )
Running phase: Value protection optimization ( 0.65s )
Error: Decompilation to LLVM IR failed

I restarted the build after that, and it failed again, but because of another test:

integration.palindrome.Test_2017 (palindrome.x86.clang.O0.g.elf)                                    [ FAIL ]  (0.85s)
F
======================================================================
FAIL: test_produce_expected_output (integration.palindrome.Test_2017)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "retdec-regression-tests/integration/palindrome/test.py", line 13, in test_produce_expected_output
    expected_output='krk is a palindrome\n'
  File "retdec-regression-tests-framework/regression_tests/tools/decompiler_test.py", line 97, in assert_c_produces_output_when_run
    self.assertEqual(actual_output, expected_output, msg)
AssertionError: 'Try again! (krk)\n' != 'krk is a palindrome\n'
- Try again! (krk)
+ krk is a palindrome
 : where the input was: krk

We either have issues with non-determinism in our tests on macOS, or there is something wrong with the macOS agent.

Possibly related issues: #267, #265, #264.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant