Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Linalg stacking tests should not test exact equality #44

Closed
leofang opened this issue Nov 24, 2021 · 3 comments
Closed

Linalg stacking tests should not test exact equality #44

leofang opened this issue Nov 24, 2021 · 3 comments

Comments

@leofang
Copy link

leofang commented Nov 24, 2021

The following two failures show that the exact equality test in _test_stacks should be relaxed, as the results agree within some tolerance that's both hardware and software dependent:

__________________________________________________________________ test_eigh __________________________________________________________________

>   ???

test_linalg.py:217: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
test_linalg.py:232: in test_eigh
    _test_stacks(lambda x: linalg.eigh(x).eigenvalues, x,
test_linalg.py:61: in _test_stacks
    assert_exactly_equal(res_stack, decomp_res_stack)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

x = Array([0.       , 1.9999999], dtype=float32), y = Array([0., 2.], dtype=float32)

    def assert_exactly_equal(x, y):
        """
        Test that the arrays x and y are exactly equal.
    
        If x and y do not have the same shape and dtype, they are not considered
        equal.
    
        """
        assert x.shape == y.shape, f"The input arrays do not have the same shapes ({x.shape} != {y.shape})"
    
        assert x.dtype == y.dtype, f"The input arrays do not have the same dtype ({x.dtype} != {y.dtype})"
    
>       assert all(exactly_equal(x, y)), "The input arrays have different values"
E       AssertionError: The input arrays have different values

array_helpers.py:181: AssertionError
----------------------------------------------------------------- Hypothesis ------------------------------------------------------------------
Falsifying example: test_eigh(
    x=Array([[[1., 1.],
            [1., 1.]]], dtype=float32),
)
________________________________________________________________ test_eigvalsh ________________________________________________________________

>   ???

test_linalg.py:241: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
test_linalg.py:248: in test_eigvalsh
    _test_stacks(linalg.eigvalsh, x, res=res, dims=1)
test_linalg.py:61: in _test_stacks
    assert_exactly_equal(res_stack, decomp_res_stack)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

x = Array([0.       , 1.9999999], dtype=float32), y = Array([0., 2.], dtype=float32)

    def assert_exactly_equal(x, y):
        """
        Test that the arrays x and y are exactly equal.
    
        If x and y do not have the same shape and dtype, they are not considered
        equal.
    
        """
        assert x.shape == y.shape, f"The input arrays do not have the same shapes ({x.shape} != {y.shape})"
    
        assert x.dtype == y.dtype, f"The input arrays do not have the same dtype ({x.dtype} != {y.dtype})"
    
>       assert all(exactly_equal(x, y)), "The input arrays have different values"
E       AssertionError: The input arrays have different values

array_helpers.py:181: AssertionError
----------------------------------------------------------------- Hypothesis ------------------------------------------------------------------
Falsifying example: test_eigvalsh(
    x=Array([[[1., 1.],
            [1., 1.]]], dtype=float32),
)
@asmeurer
Copy link
Member

The idea with the stacking tests is that a function that operates on stacks of matrices should be the same on a stacked array as if you do it on the individual arrays independently. Is there any reason this shouldn't hold? The computations are independent of one another for different matrices.

To be fair the spec doesn't really specify this. Maybe it should clarify whether it should be the same or just "close".

@leofang
Copy link
Author

leofang commented Dec 1, 2021

Sorry @asmeurer I missed your reply. What I meant is that the comparison of actual & expected outputs like this

x = Array([0.       , 1.9999999], dtype=float32), y = Array([0., 2.], dtype=float32)

should return True. In other words, we can't expect an exact comparison between floating points that is currently done here:

assert_exactly_equal(res_stack, decomp_res_stack)

@leofang
Copy link
Author

leofang commented Dec 5, 2021

cc: @kmaehashi for vis

@asmeurer asmeurer changed the title Test condition too stringent Linalg stacking tests should not test exact equality Jan 25, 2022
asmeurer added a commit to asmeurer/array-api-tests that referenced this issue Jan 25, 2022
We will eventually change this to be a float-point approximate check rather
than exact equality. For now, this is not updated.

See data-apis#44
@honno honno closed this as completed in 9bb8c7a Feb 26, 2024
cr313 added a commit to cr313/test-array-api that referenced this issue Apr 19, 2024
We will eventually change this to be a float-point approximate check rather
than exact equality. For now, this is not updated.

See data-apis/array-api-tests#44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants
@asmeurer @leofang and others