#1502 break unit tests on Address (asan) #1503

jstrzebonski · 2021-07-20T00:01:55Z

After going through some experiments it turns out that the problem was only with Open MPI on my machine. After compiling vt in docker with MPICH, all errors are gone. One important observation I made - problems found in Open MPI were reported as warnings, and I wasn't able to force tests to fail. Which might be a good thing - after all it wasn't vt's code that was problematic.

Anyway, I tried artificially make Address Sanitizer complain, and it seems that test fails, or even more - is interrupted, without any additional work, if only ASAN finds something suspicious. But just in case I leave this snippet here, it might come handy someday.

extern "C" {

void __asan_on_error() {
  FAIL() << "Encountered an address sanitizer error";
}

}

Also for testing such behavior, if ever needed, there is a gtest macro available: EXPECT_FATAL_FAILURE, and it could be used for example like this:

#include <gtest/gtest.h>
#include <gtest/gtest-spi.h>

void test_if_asan_fails() {
  auto* buffer = new char[1024];
  delete[] buffer;
  std::sprintf(
    buffer, "%d", static_cast<int>(reinterpret_cast<std::intptr_t>(&buffer))
  );
  std::printf("%s\n", buffer);
}

TEST(TestSanitizers, test_if_asan_fails) {
  EXPECT_FATAL_FAILURE(
    test_if_asan_fails(), "Encountered an address sanitizer error"
  );
}

Although as stated before, in case Asan finds anything, the test breaks, and logs with problem description, are printed.

In the end, all what this PR does is cleaning up mechanism of compiling vt with Address Sanitizer enabled.

github-actions · 2021-07-20T00:04:02Z

PR tests (intel 18.03, ubuntu, mpich)

Build for b1b73bb

Build failed for unknown reason. Check build logs

Build log

github-actions · 2021-07-20T00:05:20Z

PR tests (intel 19, ubuntu, mpich)

Build for b1b73bb

Build failed for unknown reason. Check build logs

Build log

github-actions · 2021-07-20T00:18:09Z

PR tests (gcc-6, ubuntu, mpich)

Build for 8a50fa1

Compilation - successful

Testing - passed

Build log

github-actions · 2021-07-20T00:21:17Z

PR tests (clang-5.0, ubuntu, mpich)

Build for 8a50fa1

Compilation - successful

Testing - passed

Build log

github-actions · 2021-07-20T00:21:22Z

PR tests (gcc-5, ubuntu, mpich)

Build for 8a50fa1

Compilation - successful

Testing - passed

Build log

github-actions · 2021-07-20T00:21:45Z

PR tests (clang-3.9, ubuntu, mpich)

Build for 8a50fa1

Compilation - successful

Testing - passed

Build log

github-actions · 2021-07-20T00:24:05Z

PR tests (gcc-10, ubuntu, openmpi, no LB)

Build for 8a50fa1

Compilation - successful

Testing - passed

Build log

github-actions · 2021-07-20T00:26:07Z

PR tests (gcc-9, ubuntu, mpich, zoltan)

Build for 8a50fa1

Compilation - successful

Testing - passed

Build log

github-actions · 2021-07-20T00:27:11Z

PR tests (gcc-8, ubuntu, mpich, address sanitizer)

Build for 8a50fa1

Compilation - successful

Testing - passed

Build log

github-actions · 2021-07-20T00:30:08Z

PR tests (clang-10, ubuntu, mpich)

Build for 8a50fa1

Compilation - successful

Testing - passed

Build log

github-actions · 2021-07-20T00:33:41Z

PR tests (clang-9, ubuntu, mpich)

Build for 8a50fa1

Compilation - successful

Testing - passed

Build log

github-actions · 2021-07-20T00:37:30Z

PR tests (nvidia cuda 11.0, ubuntu, mpich)

Build for 8a50fa1

Compilation - successful

Testing - passed

Build log

github-actions · 2021-07-20T00:38:12Z

PR tests (nvidia cuda 10.1, ubuntu, mpich)

Build for 8a50fa1

Compilation - successful

Testing - passed

Build log

github-actions · 2021-07-20T00:38:34Z

PR tests (gcc-7, ubuntu, mpich, trace runtime, LB)

Build for 8a50fa1

Compilation - successful

Testing - passed

Build log

codecov · 2021-07-20T00:38:37Z

Codecov Report

Merging #1503 (c42a5f1) into develop (b83a567) will increase coverage by 0.03%.
The diff coverage is n/a.

❗ Current head c42a5f1 differs from pull request most recent head 8a50fa1. Consider uploading reports for the commit 8a50fa1 to get more accurate results

@@             Coverage Diff             @@
##           develop    #1503      +/-   ##
===========================================
+ Coverage    83.03%   83.06%   +0.03%     
===========================================
  Files          784      784              
  Lines        29650    29650              
===========================================
+ Hits         24620    24630      +10     
+ Misses        5030     5020      -10

Impacted Files	Coverage Δ
src/vt/vrt/collection/manager.impl.h	`96.46% <0.00%> (+0.88%)`	⬆️
src/vt/vrt/collection/manager.h	`100.00% <0.00%> (+27.27%)`	⬆️

github-actions · 2021-07-20T00:50:25Z

PR tests (clang-10, alpine, mpich)

Build for 8a50fa1

Compilation - successful

Testing - passed

Build log

PhilMiller · 2021-08-25T21:15:59Z

Would it be possible to add a death test for one or both of these, to ensure that errors are being both caught and reported as expected?

jstrzebonski · 2021-09-16T19:58:32Z

ci/docker/ubuntu-20.04-clang-cpp.dockerfile

@@ -25,6 +25,7 @@ RUN apt-get update -y -q && \
    make-guile \
    libomp5 \
    libomp-dev \
+    llvm-10 \


In llvm-10 package there is a llvm-symbolizer. Thanks to it produced traces are human-readable, for example:

879: [ RUN ] TestSanitizers.test_if_asan_fails 879: ================================================================= 879: ==24915==ERROR: AddressSanitizer: heap-use-after-free on address 0x619000004680 at pc 0x0000012ba187 bp 0x7ffcd5045d20 sp 0x7ffcd50454b8 879: WRITE of size 11 at 0x619000004680 thread T0 879: #0 0x12ba186 in vsprintf (/build/vt/tests/utils_nompi+0x12ba186) 879: #1 0x12bb113 in sprintf (/build/vt/tests/utils_nompi+0x12bb113) 879: #2 0x136226a in vt::tests::unit::test_if_asan_fails() /vt/tests/unit/utils/test_safe_union.nompi.cc:301:3 879: #3 0x13626e8 in vt::tests::unit::TestSanitizers_test_if_asan_fails_Test::TestBody()::GTestExpectFatalFailureHelper::Execute() /vt/tests/unit/utils/test_safe_union.nompi.cc:307:3 879: #4 0x13624fe in vt::tests::unit::TestSanitizers_test_if_asan_fails_Test::TestBody() /vt/tests/unit/utils/test_safe_union.nompi.cc:307:3 879: #5 0x146946a in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /vt/tests/extern/googletest/googletest/src/gtest.cc:2433:10 879: #6 0x142d737 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /vt/tests/extern/googletest/googletest/src/gtest.cc:2469:14 879: #7 0x13e4d49 in testing::Test::Run() /vt/tests/extern/googletest/googletest/src/gtest.cc:2508:5 879: #8 0x13e6e59 in testing::TestInfo::Run() /vt/tests/extern/googletest/googletest/src/gtest.cc:2684:11 879: #9 0x13e8124 in testing::TestSuite::Run() /vt/tests/extern/googletest/googletest/src/gtest.cc:2816:28 879: #10 0x1404565 in testing::internal::UnitTestImpl::RunAllTests() /vt/tests/extern/googletest/googletest/src/gtest.cc:5338:44 879: #11 0x1471a2a in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /vt/tests/extern/googletest/googletest/src/gtest.cc:2433:10 879: #12 0x14343f2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /vt/tests/extern/googletest/googletest/src/gtest.cc:2469:14 879: #13 0x1403880 in testing::UnitTest::Run() /vt/tests/extern/googletest/googletest/src/gtest.cc:4925:10 879: #14 0x136b010 in RUN_ALL_TESTS() /vt/tests/extern/googletest/googletest/include/gtest/gtest.h:2473:46 879: #15 0x1345987 in main /vt/tests/unit/main.cc:66:10 879: #16 0x7fe2f733b0b2 in __libc_start_main /build/glibc-eX1tMB/glibc-2.31/csu/../csu/libc-start.c:308:16 879: #17 0x129b19d in _start (/build/vt/tests/utils_nompi+0x129b19d) 879: 879: 0x619000004680 is located 0 bytes inside of 1024-byte region [0x619000004680,0x619000004a80) 879: freed by thread T0 here: 879: ==24915==AddressSanitizer CHECK failed: /build/llvm-toolchain-10-yegZYJ/llvm-toolchain-10-10.0.0/compiler-rt/lib/asan/asan_descriptions.cpp:177 "((res.trace)) != (0)" (0x0, 0x0) 879: #0 0x131ba8e in __asan::AsanCheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/build/vt/tests/utils_nompi+0x131ba8e) 879: #1 0x132ffaf in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/build/vt/tests/utils_nompi+0x132ffaf) 879: #2 0x12a412b in __asan::HeapAddressDescription::Print() const (/build/vt/tests/utils_nompi+0x12a412b) 879: #3 0x12a6f75 in __asan::ErrorGeneric::Print() (/build/vt/tests/utils_nompi+0x12a6f75) 879: #4 0x13176f8 in __asan::ScopedInErrorReport::~ScopedInErrorReport() (/build/vt/tests/utils_nompi+0x13176f8) 879: #5 0x131932d in __asan::ReportGenericError(unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned long, unsigned int, bool) (/build/vt/tests/utils_nompi+0x131932d) 879: #6 0x12ba1a8 in vsprintf (/build/vt/tests/utils_nompi+0x12ba1a8) 879: #7 0x12bb113 in sprintf (/build/vt/tests/utils_nompi+0x12bb113) 879: #8 0x136226a in vt::tests::unit::test_if_asan_fails() /vt/tests/unit/utils/test_safe_union.nompi.cc:301:3 879: #9 0x13626e8 in vt::tests::unit::TestSanitizers_test_if_asan_fails_Test::TestBody()::GTestExpectFatalFailureHelper::Execute() /vt/tests/unit/utils/test_safe_union.nompi.cc:307:3 879: #10 0x13624fe in vt::tests::unit::TestSanitizers_test_if_asan_fails_Test::TestBody() /vt/tests/unit/utils/test_safe_union.nompi.cc:307:3 879: #11 0x146946a in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /vt/tests/extern/googletest/googletest/src/gtest.cc:2433:10 879: #12 0x142d737 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /vt/tests/extern/googletest/googletest/src/gtest.cc:2469:14 879: #13 0x13e4d49 in testing::Test::Run() /vt/tests/extern/googletest/googletest/src/gtest.cc:2508:5 879: #14 0x13e6e59 in testing::TestInfo::Run() /vt/tests/extern/googletest/googletest/src/gtest.cc:2684:11 879: #15 0x13e8124 in testing::TestSuite::Run() /vt/tests/extern/googletest/googletest/src/gtest.cc:2816:28 879: #16 0x1404565 in testing::internal::UnitTestImpl::RunAllTests() /vt/tests/extern/googletest/googletest/src/gtest.cc:5338:44 879: #17 0x1471a2a in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /vt/tests/extern/googletest/googletest/src/gtest.cc:2433:10 879: #18 0x14343f2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /vt/tests/extern/googletest/googletest/src/gtest.cc:2469:14 879: #19 0x1403880 in testing::UnitTest::Run() /vt/tests/extern/googletest/googletest/src/gtest.cc:4925:10 879: #20 0x136b010 in RUN_ALL_TESTS() /vt/tests/extern/googletest/googletest/include/gtest/gtest.h:2473:46 879: #21 0x1345987 in main /vt/tests/unit/main.cc:66:10 879: #22 0x7fe2f733b0b2 in __libc_start_main /build/glibc-eX1tMB/glibc-2.31/csu/../csu/libc-start.c:308:16 879: #23 0x129b19d in _start (/build/vt/tests/utils_nompi+0x129b19d) 879: 1/1 Test #879: vt:TestSanitizers.test_if_asan_fails_no_mpi ...***Failed Required regular expression not found. Regex=[PASSED ] 0.19 sec 0% tests passed, 1 tests failed out of 1

If anyone is interested, more info can be found here -> https://llvm.org/docs/CommandGuide/llvm-symbolizer.html

cmake-modules/SetCXXCompilerFlags.cmake

cz4rs

Looks good to me!

PhilMiller · 2021-09-17T16:29:33Z

The changes being made all look fine.

It would be good to add a set of suppressions for the reports generated within OpenMPI, so that we could potentially use that as well.

Also, there are runtime configuration variables that can control the sanitizers' behavior, of what counts as an error, what to do when one occurs, etc. Maybe those would play a role in carrying this further.

jstrzebonski · 2021-09-17T16:59:21Z

The changes being made all look fine.

It would be good to add a set of suppressions for the reports generated within OpenMPI, so that we could potentially use that as well.

Also, there are runtime configuration variables that can control the sanitizers' behavior, of what counts as an error, what to do when one occurs, etc. Maybe those would play a role in carrying this further.

@PhilMiller I created two issues for that purpose:

Prepare suppressions list for Address Sanitizer for vt+OpenMPI combination #1557
Investigate Address Sanitizer options in context of controlling its behavior #1558

Are you okay with merging this PR, and splitting additional work for better visibility?

PhilMiller · 2021-09-17T19:48:39Z

To be clear, we should have a test that definitively triggers the sanitizer, and assert that it reports a failure. Is that the case?

PhilMiller · 2021-09-17T19:51:45Z

I read the description of the PR to mean that with the code as is before this PR, if a test does trigger ASan, then it is reported as a failure. Is that right?

jstrzebonski · 2021-09-21T15:42:58Z

I read the description of the PR to mean that with the code as is before this PR, if a test does trigger ASan, then it is reported as a failure. Is that right?

@PhilMiller Yes.

… usage

jstrzebonski self-assigned this Jul 20, 2021

PhilMiller changed the title ~~#1502 break unit tests on ASAN errors~~ #1502 break unit tests on Address (asan) and Undefined Behavior (ubsan) Sanitizer errors Aug 25, 2021

jstrzebonski force-pushed the 1502-break-unit-tests-on-sanitizer-error branch from b1b73bb to 425e9e7 Compare September 16, 2021 18:51

jstrzebonski changed the title ~~#1502 break unit tests on Address (asan) and Undefined Behavior (ubsan) Sanitizer errors~~ #1502 break unit tests on Address (asan) Sep 16, 2021

jstrzebonski commented Sep 16, 2021

View reviewed changes

jstrzebonski force-pushed the 1502-break-unit-tests-on-sanitizer-error branch from 425e9e7 to 3fd8d91 Compare September 16, 2021 20:56

jstrzebonski requested review from lifflander, cz4rs and JacobDomagala September 17, 2021 15:04

jstrzebonski marked this pull request as ready for review September 17, 2021 15:04

cz4rs reviewed Sep 17, 2021

View reviewed changes

cmake-modules/SetCXXCompilerFlags.cmake Outdated Show resolved Hide resolved

cz4rs approved these changes Sep 17, 2021

View reviewed changes

jstrzebonski requested a review from PhilMiller September 17, 2021 16:28

PhilMiller approved these changes Sep 21, 2021

View reviewed changes

Jakub Strzebonski added 8 commits September 21, 2021 19:25

#1502 break unit tests on ASAN errors

Unverified

No user is associated with the committer email.

Learn about vigilant mode

c7b3772

#1502 remove redundant sanitizers reporting functions

Unverified

No user is associated with the committer email.

Learn about vigilant mode

a83a8e6

#1502 rename suppression file to better point sanitizer it affects

Unverified

No user is associated with the committer email.

Learn about vigilant mode

f6727aa

#1502 add llvm-10 to 'Ubuntu 20.04 + clang-10' dockerfile

a0aeba3

#1502 remove redundat compiler check

a0e5fbb

#1502 change compiler flags to ones recommended for Address Sanitizer…

ab317c0

… usage

#1502 fix typo in file name

90a6fd8

#1502 remove some lost experiments leftovers

8a50fa1

jstrzebonski force-pushed the 1502-break-unit-tests-on-sanitizer-error branch from c42a5f1 to 8a50fa1 Compare September 21, 2021 17:25

jstrzebonski merged commit cf546ba into develop Sep 28, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#1502 break unit tests on Address (asan) #1503

#1502 break unit tests on Address (asan) #1503

jstrzebonski commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021

github-actions bot commented Jul 20, 2021

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

codecov bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

PhilMiller commented Aug 25, 2021

jstrzebonski Sep 16, 2021 •

edited

Loading

cz4rs left a comment

PhilMiller commented Sep 17, 2021

jstrzebonski commented Sep 17, 2021

PhilMiller commented Sep 17, 2021

PhilMiller commented Sep 17, 2021

jstrzebonski commented Sep 21, 2021

#1502 break unit tests on Address (asan) #1503

#1502 break unit tests on Address (asan) #1503

Conversation

jstrzebonski commented Jul 20, 2021 • edited Loading

github-actions bot commented Jul 20, 2021

github-actions bot commented Jul 20, 2021

github-actions bot commented Jul 20, 2021 • edited Loading

github-actions bot commented Jul 20, 2021 • edited Loading

github-actions bot commented Jul 20, 2021 • edited Loading

github-actions bot commented Jul 20, 2021 • edited Loading

github-actions bot commented Jul 20, 2021 • edited Loading

github-actions bot commented Jul 20, 2021 • edited Loading

github-actions bot commented Jul 20, 2021 • edited Loading

github-actions bot commented Jul 20, 2021 • edited Loading

github-actions bot commented Jul 20, 2021 • edited Loading

github-actions bot commented Jul 20, 2021 • edited Loading

github-actions bot commented Jul 20, 2021 • edited Loading

github-actions bot commented Jul 20, 2021 • edited Loading

codecov bot commented Jul 20, 2021 • edited Loading

Codecov Report

github-actions bot commented Jul 20, 2021 • edited Loading

PhilMiller commented Aug 25, 2021

jstrzebonski Sep 16, 2021 • edited Loading

Choose a reason for hiding this comment

cz4rs left a comment

Choose a reason for hiding this comment

PhilMiller commented Sep 17, 2021

jstrzebonski commented Sep 17, 2021

PhilMiller commented Sep 17, 2021

PhilMiller commented Sep 17, 2021

jstrzebonski commented Sep 21, 2021

jstrzebonski commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

codecov bot commented Jul 20, 2021 •

edited

Loading

github-actions bot commented Jul 20, 2021 •

edited

Loading

jstrzebonski Sep 16, 2021 •

edited

Loading