Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Flaky Test] tests/python/frontend/pytorch/test_forward.py::test_empty_like #11967

Closed
driazati opened this issue Jun 30, 2022 · 5 comments · Fixed by #11890
Closed

[Flaky Test] tests/python/frontend/pytorch/test_forward.py::test_empty_like #11967

driazati opened this issue Jun 30, 2022 · 5 comments · Fixed by #11890

Comments

@driazati
Copy link
Member

driazati commented Jun 30, 2022

Test tests/python/frontend/pytorch/test_forward.py::test_empty_like https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/3713/tests is flaky and has failed intermittently on main (as discussed in #11896)

[2022-06-29T08:34:53.751Z] _______________________________ test_empty_like ________________________________
[2022-06-29T08:34:53.751Z] 
[2022-06-29T08:34:53.751Z]     def test_empty_like():
[2022-06-29T08:34:53.751Z]         def test_func(data):
[2022-06-29T08:34:53.751Z]             return torch.empty_like(data)
[2022-06-29T08:34:53.751Z]     
[2022-06-29T08:34:53.751Z] >       verify_model_with_input(test_func, [torch.rand([1, 3, 10, 10]).float()])
[2022-06-29T08:34:53.751Z] 
[2022-06-29T08:34:53.751Z] tests/python/frontend/pytorch/test_forward.py:3827: 
[2022-06-29T08:34:53.751Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[2022-06-29T08:34:53.751Z] tests/python/frontend/pytorch/test_forward.py:221: in verify_model_with_input
[2022-06-29T08:34:53.751Z]     tvm.testing.assert_allclose(baseline_outputs, compiled_output, rtol=1e-5, atol=1e-5)
[2022-06-29T08:34:53.751Z] python/tvm/testing/utils.py:113: in assert_allclose
[2022-06-29T08:34:53.751Z]     np.testing.assert_allclose(actual, desired, rtol=rtol, atol=atol, verbose=True)
[2022-06-29T08:34:53.751Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[2022-06-29T08:34:53.751Z] 
[2022-06-29T08:34:53.751Z] x = array([[[[ 7.7170313e-17,  0.0000000e+00,  5.3485164e-26,
[2022-06-29T08:34:53.751Z]            0.0000000e+00,  0.0000000e+00,  0.0000000e+00,
[2022-06-29T08:34:53.751Z]   ...  0.0000000e+00,
[2022-06-29T08:34:53.751Z]            0.0000000e+00,  0.0000000e+00,  0.0000000e+00,
[2022-06-29T08:34:53.751Z]            0.0000000e+00]]]], dtype=float32)
[2022-06-29T08:34:53.751Z] y = array([[[[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[2022-06-29T08:34:53.751Z]          [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[2022-06-29T08:34:53.751Z]          [0., 0....         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[2022-06-29T08:34:53.751Z]          [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]]], dtype=float32)
[2022-06-29T08:34:53.751Z] func = <ufunc 'isnan'>, hasval = 'nan'
[2022-06-29T08:34:53.751Z] 

From the diff there doesn't appear to be an actual difference but we should disable anyways to not block while its figured out

cc @shingjan

@driazati driazati changed the title [ci] [Flaky Test] tests/python/frontend/pytorch/test_forward.py::test_empty_like Jun 30, 2022
driazati added a commit to driazati/tvm that referenced this issue Jun 30, 2022
@shingjan
Copy link

shingjan commented Jun 30, 2022

@driazati It's possible that empty is flaky as well bcoz it shares a same implementation virtually with empty_like. vvv Altan's comments

@altanh
Copy link
Contributor

altanh commented Jun 30, 2022

empty_like ostensibly returns uninitialized memory so we really shouldn't test this numerically (https://pytorch.org/docs/stable/generated/torch.empty_like.html)

@shingjan
Copy link

@altanh What about we instead test if the outputs are of the same shape under the condition that torch.empty_like may return uninitialized memory in this case.

masahi pushed a commit that referenced this issue Jun 30, 2022
@masahi
Copy link
Member

masahi commented Jul 1, 2022

I agree that we should only verify the output shape.

@shingjan
Copy link

shingjan commented Jul 1, 2022

#11890 is out to fix this.

blackkker pushed a commit to blackkker/tvm that referenced this issue Jul 7, 2022
masahi pushed a commit to masahi/tvm that referenced this issue Jul 15, 2022
mikeseven pushed a commit to mikeseven/tvm that referenced this issue Sep 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants