Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

write to zarr fails when SpatialImage is created from a xarray.DataArray #59

Closed
LucaMarconato opened this issue Nov 30, 2022 · 4 comments · Fixed by #127
Closed

write to zarr fails when SpatialImage is created from a xarray.DataArray #59

LucaMarconato opened this issue Nov 30, 2022 · 4 comments · Fixed by #127
Assignees

Comments

@LucaMarconato
Copy link
Member

When the SpatialImage is created from a np.ndarray all is good, when is created from a xarray.DataArray an error occurs (see traceback).
I have added tests for this. To reproduce uncomment the parts of _get_images() and _get_labels() (from tests/conftest.py) that start with out['....._xarray'] = ......

tests/_io/test_readwrite.py:14 (TestReadWrite.test_images)
self = <tests._io.test_readwrite.TestReadWrite object at 0x7f9f9490edd0>
tmp_path = PosixPath('/private/tmp/pytest-of-macbook/pytest-74/test_images0')
images = SpatialData object with:
└── Images
│     ├── 'image2d_numpy': SpatialImage[cyx] (3, 64, 64)
│     ├── 'image2d_multis...3, 64, 64)
│     └── 'image2d_multiscale_numpy_xarray': MultiscaleSpatialImage[cyx] (3, 64, 64), (3, 32, 32), (3, 8, 8)

    def test_images(self, tmp_path: str, images: SpatialData) -> None:
        """Test read/write."""
        tmpdir = Path(tmp_path) / "tmp.zarr"
>       images.write(tmpdir)

tests/_io/test_readwrite.py:18: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
spatialdata/_core/_spatialdata.py:147: in write
    write_image(
spatialdata/_io/write.py:95: in write_image
    write_image_ngff(
../../../../miniconda3/envs/ome/lib/python3.10/site-packages/ome_zarr/writer.py:455: in write_image
    _write_dask_image(
../../../../miniconda3/envs/ome/lib/python3.10/site-packages/ome_zarr/writer.py:565: in _write_dask_image
    da.compute(*delayed)
../../../../miniconda3/envs/ome/lib/python3.10/site-packages/dask/base.py:600: in compute
    results = schedule(dsk, keys, **kwargs)
../../../../miniconda3/envs/ome/lib/python3.10/site-packages/dask/threaded.py:89: in get
    results = get_async(
../../../../miniconda3/envs/ome/lib/python3.10/site-packages/dask/local.py:511: in get_async
    raise_exception(exc, tb)
../../../../miniconda3/envs/ome/lib/python3.10/site-packages/dask/local.py:319: in reraise
    raise exc
../../../../miniconda3/envs/ome/lib/python3.10/site-packages/dask/local.py:224: in execute_task
    result = _execute_task(task, data)
../../../../miniconda3/envs/ome/lib/python3.10/site-packages/dask/core.py:119: in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
../../../../miniconda3/envs/ome/lib/python3.10/site-packages/dask/array/core.py:4366: in store_chunk
    return load_store_chunk(x, out, index, lock, return_stored, False)
../../../../miniconda3/envs/ome/lib/python3.10/site-packages/dask/array/core.py:4348: in load_store_chunk
    out[index] = x
../../../../miniconda3/envs/ome/lib/python3.10/site-packages/zarr/core.py:1373: in __setitem__
    self.set_basic_selection(pure_selection, value, fields=fields)
../../../../miniconda3/envs/ome/lib/python3.10/site-packages/zarr/core.py:1468: in set_basic_selection
    return self._set_basic_selection_nd(selection, value, fields=fields)
../../../../miniconda3/envs/ome/lib/python3.10/site-packages/zarr/core.py:1772: in _set_basic_selection_nd
    self._set_selection(indexer, value, fields=fields)
../../../../miniconda3/envs/ome/lib/python3.10/site-packages/zarr/core.py:1844: in _set_selection
    self._chunk_setitems(lchunk_coords, lchunk_selection, chunk_values,
../../../../miniconda3/envs/ome/lib/python3.10/site-packages/zarr/core.py:2045: in _chunk_setitems
    to_store = {k: self._encode_chunk(v) for k, v in cdatas.items()}
../../../../miniconda3/envs/ome/lib/python3.10/site-packages/zarr/core.py:2045: in <dictcomp>
    to_store = {k: self._encode_chunk(v) for k, v in cdatas.items()}
../../../../miniconda3/envs/ome/lib/python3.10/site-packages/zarr/core.py:2222: in _encode_chunk
    if ensure_ndarray_like(chunk).dtype == object:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

buf = <xarray.DataArray (c: 3, y: 64, x: 64)>
array([[[-1.45060918e+00,  5.93752346e-01, -3.55374641e-02, ...,
         -5.1...053254e+00, ...,
         -2.45753512e-01,  1.03229240e+00, -8.63332273e-01]]])
Dimensions without coordinates: c, y, x

    def ensure_ndarray_like(buf) -> NDArrayLike:
        """Convenience function to coerce `buf` to ndarray-like array.
    
        Parameters
        ----------
        buf : ndarray-like, array-like, or bytes-like
            A numpy array like object such as numpy.ndarray, cupy.ndarray, or
            any object exporting a buffer interface.
    
        Returns
        -------
        arr : NDArrayLike
            A ndarray-like, sharing memory with `buf`.
    
        Notes
        -----
        This function will not create a copy under any circumstances, it is guaranteed to
        return a view on memory exported by `buf`.
        """
    
        if not is_ndarray_like(buf):
            if isinstance(buf, array.array) and buf.typecode in "cu":
                # Guard condition, do not support array.array with unicode type, this is
                # problematic because numpy does not support it on all platforms. Also do not
                # support char as it was removed in Python 3.
                raise TypeError("array.array with char or unicode type is not supported")
            else:
                # N.B., first take a memoryview to make sure that we subsequently create a
                # numpy array from a memory buffer with no copy
>               mem = memoryview(buf)
E               TypeError: memoryview: a bytes-like object is required, not 'DataArray'

../../../../miniconda3/envs/ome/lib/python3.10/site-packages/numcodecs/compat.py:42: TypeError
@giovp
Copy link
Member

giovp commented Nov 30, 2022

I believe it has to do with the data type (xarray with numpy array might be a problem). Will check!

@LucaMarconato
Copy link
Member Author

LucaMarconato commented Nov 30, 2022

In my previous more experimental branch (#48) I was converting everything to DataArray instead than to Dask arrays. This didn't induce the problem, and Dask arrays converted to DataArray where still backed. I tried changing this in the new code but then I noticed that the Schema asks for Dask arrays, so I didn't want to change that before checking with you. Maybe this could fix it.

@giovp
Copy link
Member

giovp commented Dec 15, 2022

I believe this is now tested and addressed in #52

@giovp giovp closed this as completed Dec 15, 2022
@LucaMarconato
Copy link
Member Author

I checked, the tests that failed are still commented in tests/conftest.py. The one added in test_models.py may still not cover those cases because maybe Dask doesn't reach the problem until things are computed/written to disk. I'll take care of this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants