CI: tests hang during test collection stage

[leezu]

Description

[2020-05-29T18:49:28.268Z] + pytest --durations=50 --cov-report xml:tests_unittest.xml --verbose tests/python/unittest

[2020-05-29T18:49:31.528Z] ============================= test session starts ==============================

[2020-05-29T18:49:31.528Z] platform linux -- Python 3.6.9, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /usr/bin/python3

[2020-05-29T18:49:31.528Z] cachedir: .pytest_cache

[2020-05-29T18:49:31.528Z] rootdir: /work/mxnet, inifile: pytest.ini

[2020-05-29T18:49:31.528Z] plugins: flaky-3.6.1, xdist-1.31.0, timeout-1.3.4, cov-2.8.1, env-0.6.2, forked-1.1.3

[2020-05-29T18:49:31.528Z] timeout: 1200.0s

[2020-05-29T18:49:31.528Z] timeout method: signal

[2020-05-29T18:49:31.528Z] timeout func_only: False

[2020-05-29T22:44:00.825Z] Sending interrupt signal to process

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-18391/1/pipeline/366

or

[2020-05-21T18:58:56.773Z] + CI_CUDA_COMPUTE_CAPABILITIES='-gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_70,code=sm_70'

[2020-05-21T18:58:56.773Z] + CI_CMAKE_CUDA_ARCH='5.2 7.0'

[2020-05-21T18:58:56.773Z] + set +x

[2020-05-21T18:58:56.773Z] + export PYTHONPATH=./python/

[2020-05-21T18:58:56.773Z] + PYTHONPATH=./python/

[2020-05-21T18:58:56.773Z] + export DMLC_LOG_STACK_TRACE_DEPTH=10

[2020-05-21T18:58:56.773Z] + DMLC_LOG_STACK_TRACE_DEPTH=10

[2020-05-21T18:58:56.773Z] + python3 tests/python/unittest/onnx/backend_test.py

[2020-05-21T22:52:22.501Z] Sending interrupt signal to process

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/master/1969/pipeline/376

[leezu]

This hasn't been fixed yet (or regressed).

[2020-10-05T19:27:03.429Z] 2020-10-05 19:27:02,234 - root - INFO - Executing the equivalent of:

[2020-10-05T19:27:03.429Z] docker \

[2020-10-05T19:27:03.429Z] 	run \

[2020-10-05T19:27:03.429Z] 	 \

[2020-10-05T19:27:03.429Z] 	--cap-add \

[2020-10-05T19:27:03.429Z] 	SYS_PTRACE \

[2020-10-05T19:27:03.429Z] 	--rm \

[2020-10-05T19:27:03.429Z] 	--shm-size=500m \

[2020-10-05T19:27:03.429Z] 	-v \

[2020-10-05T19:27:03.429Z] 	/home/jenkins_slave/workspace/it-onnx-cpu:/work/mxnet \

[2020-10-05T19:27:03.429Z] 	-v \

[2020-10-05T19:27:03.429Z] 	/home/jenkins_slave/workspace/it-onnx-cpu/build:/work/build \

[2020-10-05T19:27:03.429Z] 	-v \

[2020-10-05T19:27:03.429Z] 	/home/jenkins_slave/.ccache:/work/ccache \

[2020-10-05T19:27:03.429Z] 	-u \

[2020-10-05T19:27:03.429Z] 	1001:1001 \

[2020-10-05T19:27:03.429Z] 	-e \

[2020-10-05T19:27:03.429Z] 	CCACHE_MAXSIZE=500G \

[2020-10-05T19:27:03.429Z] 	-e \

[2020-10-05T19:27:03.429Z] 	CCACHE_TEMPDIR=/tmp/ccache \

[2020-10-05T19:27:03.429Z] 	-e \

[2020-10-05T19:27:03.429Z] 	CCACHE_DIR=/work/ccache \

[2020-10-05T19:27:03.429Z] 	-e \

[2020-10-05T19:27:03.429Z] 	CCACHE_LOGFILE=/tmp/ccache.log \

[2020-10-05T19:27:03.429Z] 	-ti \

[2020-10-05T19:27:03.429Z] 	mxnetci/build.ubuntu_cpu:latest \

[2020-10-05T19:27:03.429Z] 	/work/runtime_functions.sh \

[2020-10-05T19:27:03.429Z] 	integrationtest_ubuntu_cpu_onnx

[2020-10-05T19:27:03.429Z] 

[2020-10-05T19:27:03.429Z] 2020-10-05 19:27:02,715 - root - INFO - Started container: f6dc093513b1

[2020-10-05T19:27:03.429Z] + CI_CUDA_COMPUTE_CAPABILITIES='-gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_70,code=sm_70'

[2020-10-05T19:27:03.429Z] + CI_CMAKE_CUDA_ARCH='5.2 7.0'

[2020-10-05T19:27:03.429Z] + set +x

[2020-10-05T19:27:03.429Z] + export PYTHONPATH=./python/

[2020-10-05T19:27:03.429Z] + PYTHONPATH=./python/

[2020-10-05T19:27:03.429Z] + export DMLC_LOG_STACK_TRACE_DEPTH=10

[2020-10-05T19:27:03.429Z] + DMLC_LOG_STACK_TRACE_DEPTH=10

[2020-10-05T19:27:03.429Z] + python3 tests/python/unittest/onnx/backend_test.py

[2020-10-05T19:27:06.689Z] [19:27:06] /work/mxnet/src/storage/storage.cc:199: Using Pooled (Naive) StorageManager for CPU

[2020-10-05T23:25:03.587Z] Sending interrupt signal to process

[2020-10-05T23:25:04.144Z] 2020-10-05 23:25:03,577 - root - WARNING - Signal 15 received, cleaning up...

[2020-10-05T23:25:04.144Z] 2020-10-05 23:25:03,578 - root - WARNING - Cleaning up containers

[2020-10-05T23:25:07.402Z] 2020-10-05 23:25:07,169 - root - INFO - ☠: stopped container f6dc093513b1

[2020-10-05T23:25:07.402Z] 2020-10-05 23:25:07,307 - root - INFO - 🚽: removed container f6dc093513b1

[2020-10-05T23:25:07.402Z] 2020-10-05 23:25:07,307 - root - INFO - Cleaning up containers finished.

[2020-10-05T23:25:07.402Z] 2020-10-05 23:25:07,307 - root - WARNING - done. Exiting with error.

[2020-10-05T23:25:07.405Z] script returned exit code 1

https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-19293/1/pipeline/287