Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix for base arrays with negative strides #949

Merged

Conversation

eslavich
Copy link
Contributor

@eslavich eslavich commented Mar 22, 2021

Here's my second attempt at a fix for #916. I noticed that we have a precedent where we copy the view when it has strides unsupported by ASDF (in the existing case, 0 strides). Since re-computing the view against the base array as serialized is hard, I propose that we do the same here and simply copy the view when its base is non-contiguous. I also added scipy to our test dependencies so that we can test this using the example in the issue description.

Fixes #916

@CagtayFabry
Copy link
Contributor

IIRC @marscher already committed a fix for the unusual stride layout created by Rotation that should make it to the next scipy release (targeted for 1.7)

In that case I am not sure if the test based on Rotation can reliably be used to induce the desired behavior in the future.

Do you agree @marscher?

@marscher
Copy link

marscher commented Mar 23, 2021 via email

@eslavich
Copy link
Contributor Author

These types of arrays simply can be produced by creating a reverse view on the memory, e.g. [::-1]

Unfortunately (or rather, fortunately!) existing releases of asdf handle this array just fine:

import asdf
import numpy as np
from numpy.testing import assert_array_equal

array = np.arange(1000)[::-1]
with asdf.AsdfFile({"array": array}) as af:
    af.write_to("test.asdf")

with asdf.open("test.asdf") as af:
    assert_array_equal(af["array"], array)

That gives a view with negative strides over a contiguous base array. The scipy array is special because the base array itself has negative strides, which means that the memory allocated for the base is laid out in reverse order from what the base array needs. As far as I can tell, that's not possible to achieve in pure Python.

In any case, thanks for the warning about the scipy fix. Do you think I should remove the test, or forego this PR entirely? If this situation is considered to be a scipy bug, then maybe it's not something that asdf needs to handle.

@marscher
Copy link

I think this edge case can only be enforced by using Cythons memoryviews and should be super rare. As long as you do not want to require Cython for your testsuite I suggest it is kind of overkill to support it.

@eslavich
Copy link
Contributor Author

Ok, I'll remove the test, but keep the fix in place.

@eslavich eslavich force-pushed the AL-484-base-arrays-with-negative-strides branch from 64f3f92 to 5a13798 Compare March 23, 2021 18:43
@eslavich eslavich requested a review from jdavies-st March 23, 2021 18:46
@eslavich eslavich force-pushed the AL-484-base-arrays-with-negative-strides branch from 5a13798 to c275f71 Compare March 23, 2021 18:46
@eslavich eslavich added this to the 2.7.4 milestone Mar 23, 2021
@eslavich eslavich force-pushed the AL-484-base-arrays-with-negative-strides branch from c275f71 to 5f78423 Compare March 23, 2021 18:49
Copy link
Contributor

@jdavies-st jdavies-st left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but without a test, I can't see what it is fixing.

@eslavich eslavich merged commit 8ce7a14 into asdf-format:master Mar 23, 2021
@eslavich eslavich deleted the AL-484-base-arrays-with-negative-strides branch March 23, 2021 19:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Scipy 1.6.0 generated ndarray can not be restored
4 participants