-
Notifications
You must be signed in to change notification settings - Fork 6.8k
[CI] Python2: CPU - hangs after test_create_np_param #16831
Comments
I have also seen a CI hang on a numpy test with the Python2 CI runner: The last test to add to the above log was test_numpy_op.py:test_numpy_reshape. The next test would have been test_numpy_resize. Have any of the developers of the new numpy facility seen hangs on numpy tests? |
It's probably due to too many logging messages dumped in the unit test. This problem has been fixed in the master branch. Could you rebase your PRs to reduce test time? |
@reminisce should the fix be backported to 1.6 branch? |
I was under the impression that when a PR goes through CI, the code tested is a merge of the PR with the then-current master. My hang was seen just 24 hours ago. @reminisce , what commit to master solved this problem in your view? |
The hang I reported in this old issue has occurred again in the exact same place, which I inferred to be in test_np_resize() because that is the next test to run after test_np_reshape(): A retry run I launched also appears to be hung on the test that follows test_np_empty(), namely test_np_empty_like(): I'm not sure how the problem is related to log file length. The lengths of the failing log files are similar, but shorter than a passing run:
|
and so on until the job is stopped after 4 hours.
http://jenkins.mxnet-ci.amazon-ml.com/blue/rest/organizations/jenkins/pipelines/mxnet-validation/pipelines/unix-cpu/branches/PR-16810/runs/7/nodes/294/steps/806/log/?start=0
The text was updated successfully, but these errors were encountered: