MPI Access Guard Death test crashes while trying to abort on error

[PhilMiller]

Describe the bug

https://dev.azure.com/DARMA-tasking/DARMA/_build/results?buildId=20700&view=logs&j=3dc8fd7e-4368-5a92-293e-d53cefc8c4b3&t=28db5144-7e5d-5c90-2820-8676d630d9d2

        Start 434: vt:TestMpiAccessGuardDeathTest.test_mpi_access_prevented_proc_2
434/605 Test #434: vt:TestMpiAccessGuardDeathTest.test_mpi_access_prevented_proc_2 ....................***Failed  Error regular expression found in output. Regex=[Segmentation fault]  0.25 sec
Note: Google Test filter = TestMpiAccessGuardDeathTest.test_mpi_access_prevented
Note: Google Test filter = TestMpiAccessGuardDeathTest.test_mpi_access_prevented
[==========] Running 1 test from 1 test suite.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from TestMpiAccessGuardDeathTest
[----------] Global test environment set-up.
[ RUN      ] TestMpiAccessGuardDeathTest.test_mpi_access_prevented
[----------] 1 test from TestMpiAccessGuardDeathTest
[ RUN      ] TestMpiAccessGuardDeathTest.test_mpi_access_prevented
vt: Runtime Initializing: interop=true: mode: single-thread per rank
vt: Program: runtime_basic (./runtime_basic)
vt: Running on: 2 nodes
vt: Machine Hostname: 5db540777f6f
vt: MPI Version: 3.1
vt: MPI Max tag: 268435455
vt: Build SHA: d42afd6ba61be7932bed40e73dbc193389e208c7
vt: Build Ref: 
vt: Description: remotes/pull/1517/merge-0-gd42afd6ba6 
vt: Compile-time Features Enabled:
vt: 	Load Balancing for Collections
vt: 	Production Build (assertions and debug prints disabled)
vt: 	Message priorities
vt: 	Memory Pooling
vt: 	MPI access guards
vt: 	Performance analysis diagnostics
vt: Runtime Configuration:
vt: 	Option: flag --vt_sched_num_progress on: Running MPI progress 2 times each invocation
vt: 	Option: flag --vt_sched_progress_han on: Running MPI progress function at least every 0 handler(s) executed
vt: 	Option: flag --vt_print_no_progress on: Printing warnings when progress is stalls
vt: 	Default: Printing verbose epoch graphs when hang detected, use --vt_epoch_graph_terse to disable
vt: 	Option: flag --vt_epoch_graph_on_hang on: Epoch graph output enabled if hang detected
vt: 	Default: Termination hang detection enabled by default, use --vt_no_detect_hang to disable
vt: 	Option: flag --vt_hang_detect on: Printing stall warning every 1024 tree traversals 
vt: 	Default: SIGINT signal handling enabled by default, use --vt_no_SIGINT to disable
vt: 	Default: SIGSEGV signal handling enabled by default, use --vt_no_SIGSEGV to disable
vt: 	Default: SIGBUS signal handling enabled by default, use --vt_no_SIGBUS to disable
vt: 	Default: std::terminate signal handling enabled by default, use --vt_no_terminate to disable
vt: 	Default: Color output enabled, use --vt_no_color to disable
vt: 	Default: Stack dumps enabled by default, use --vt_no_stack to disable
vt: 	Option: flag --vt_memory_reporters on: Memory usage checker precedence: mstats,machinfo,selfstat,selfstatm,sbrk,mallinfo,getrusage,ps
vt: 	Working memory reporters: selfstatm,getrusage
vt: 	Initial memory usage: selfstatm=14.7812 getrusage=14.7812 (MiB)
vt: 	Default: Memory usage printing disabled, use --vt_print_memory_each_phase to enable
vt: 	Default: Diagnostics are disabled by default, use --vt_diag_enable to enable
vt: 	Option: flag --vt_max_mpi_send_size on: Splitting messages greater than 1 GiB
vt: 	Option: flag --vt_debug_level on: Debug print mode set to terse
vt: Pass-through Arguments:
vt: 	None. All arguments handled.
vt: [1] ------------------------------------------------------------------------------------------------------------------------
vt: [1] ------------------------------------------- Runtime Error: System Aborting! --------------------------------------------
vt: [1] ------------------------------------------------ Fatal Error on Node 1 -------------------------------------------------
vt: [1] ------------------------------------------------------------------------------------------------------------------------
vt: [1] 
vt: [1] Message: FATAL ERROR: The MPI function 'MPI_Test' was called from a VT handler. MPI functions should not used inside user code invoked from VT handlers.

vt: [1] 
vt: Caught SIGSEGV signal: 11 

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 14476 RUNNING AT 5db540777f6f
=   EXIT CODE: 139
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions

To Reproduce
Steps to reproduce the behavior:

1. Example/test/snippet of code that fails
2. Compiler, platform, libraries
3. Run command: number of processors, threading options, etc.
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Platform (please complete the following information):

• Supercomputer or desktop [e.g., Trinity, Sierra)
• Compiler environment [e.g., clang-3.9, gcc8]
• Libraries [e.g., mpich-2, openmpi)
• Commit ID or release of VT [e.g., vt-1.0.0]

Additional context
Add any other context about the problem here.

[JacobDomagala]

This only happens on Alpine image and (from what I remember) the segfault happens when vt tries to print the backtrace. This seems to be a common issue on Alpine (mikroskeem/libexecinfo#2).

We discussed this a few weeks ago and decided we're no longer going to use Alpine image, instead we're going to switch to Red Hat/CentOS (see #1495)

[lifflander]

I'm closing this as we are moving to CentOS so this shouldn't be an issue in the future.