Running stdlib in a loop can hang

[winksaville]

I ran into this hang on my Arch Linux system running on LLVM 5.0. The code is in branch support-openssl-1.1-simple-travis
of my ponyc fork. This is obviously non-standard code and an unsupported configuration, but I believe its worth while investigating why the hang is occuring. Note: I ran across this testing #2297 which I'm now looking at in parallel.

I build ponyc and stdlib with:

make default_pic=true test-stdlib-debug

And then ran stdlib in the following loop. And after about 30 minutes it hung:

while true; do DATE=`date`; echo "Date: ${DATE}"; ./stdlib --sequential; echo "stdlib done"; done 2>&1 | tee stdlib-171223-1615.txt
Date: Sat Dec 23 18:15:13 PST 2017
run_thread:+ tid=140686117254912
run_thread:+ tid=140685957871360
run_thread:+ tid=140686259861248
run_thread:+ tid=140686100477696
ponyint_sched_shutdown:+
ponyint_sched_shutdown: join i=0 tid=140686259861248
run_thread:+ tid=140686092089088
run_thread:+ tid=140686108866304
_TestRunner.create:+ id=0
_TestRunner.create:+ id=2
_TestRunner.create:+ id=3
_TestRunner.create:- id=0
_TestRunner.create:- id=3
_TestRunner.create:+ id=1
_TestRunner.create:- id=1
...
�[92m---- Passed: itertools/Iter.run�[0m
�[92m---- Passed: itertools/Iter.skip�[0m
�[92m---- Passed: itertools/Iter.skip_while�[0m
�[92m---- Passed: itertools/Iter.take�[0m
�[92m---- Passed: itertools/Iter.take_while�[0m
�[92m---- Passed: itertools/Iter.zip�[0m
�[92m---- Passed: serialise/Simple�[0m
�[92m---- Passed: serialise/Arrays�[0m
�[92m---- Passed: serialise/Failures�[0m
�[92m---- Passed: serialise/BoxedMachineWord�[0m
----
---- 277 tests ran.
�[92m---- Passed: 277�[0m
run_thread: 1 tid=140355670398720
run_thread: 1 tid=140355829782272
run_thread:- tid=140355670398720
run_thread:- tid=140355829782272
run_thread: 1 tid=140355821393664
run_thread:- tid=140355821393664
run_thread: 1 tid=140355813005056
run_thread:- tid=140355813005056

Download stdlib-171223-1615.txt.tar.bz2. The uncompressed file contains 8M lines and is 283M bytes in size.

I also tested with pthreads scheduler:

make default_pic=true use=scheduler_scaling_pthreads test-stdlib-debug

The following bash loop was used for the scheduler_scaling_pthreads test which hung after about
12 minutes it hung:

while true; do DATE=`date`; echo "Date: ${DATE}"; ./stdlib --sequential; echo "stdlib done"; done 2>&1 | tee stdlib-171223-2044.txt
Date: Sat Dec 23 20:44:21 PST 2017
run_thread:+ tid=140496210622208
run_thread:+ tid=140495908632320
run_thread:+ tid=140496068015872
run_thread:+ tid=140496059627264
ponyint_sched_shutdown:+
ponyint_sched_shutdown: join i=0 tid=140496210622208
run_thread:+ tid=140496051238656
run_thread:+ tid=140496042850048
_TestRunner.create:+ id=0
_TestRunner.create:+ id=1
_TestRunner.create:- id=1
_TestRunner.create:- id=0
_TestRunner.run:+ id=0
_TestRunner.create:+ id=3
_TestRunner.create:- id=3
_TestRunner.create:+ id=2
---
�[92m---- Passed: itertools/Iter.fold�[0m
�[92m---- Passed: itertools/Iter.last�[0m
�[92m---- Passed: itertools/Iter.map�[0m
�[92m---- Passed: itertools/Iter.nth�[0m
�[92m---- Passed: itertools/Iter.run�[0m
�[92m---- Passed: itertools/Iter.skip�[0m
�[92m---- Passed: itertools/Iter.skip_while�[0m
�[92m---- Passed: itertools/Iter.take�[0m
�[92m---- Passed: itertools/Iter.take_while�[0m
�[92m---- Passed: itertools/Iter.zip�[0m
�[92m---- Passed: serialise/Simple�[0m
�[92m---- Passed: serialise/Arrays�[0m
�[92m---- Passed: serialise/Failures�[0m
�[92m---- Passed: serialise/BoxedMachineWord�[0m
----
---- 277 tests ran.
�[92m---- Passed: 277�[0m
run_thread: 1 tid=140152762701568
run_thread:- tid=140152762701568
run_thread: 1 tid=140152779478784
run_thread: 1 tid=140152620095232
run_thread:- tid=140152779478784
run_thread:- tid=140152620095232
run_thread: 1 tid=140152771090176
run_thread:- tid=140152771090176

Dlownload stdlib-171223-2044.txt.tar.bz2. The uncompressed file contains 3M lines and is 131M bytes in size.