DataArray to_dict() without converting with numpy tolist()

[nicain]

Enhancement:
Following #432 and #917, I need to convert my DataArray to a dictionary, but do not need to fully-jsonize the data. In fact, I would much prefer to get the data without using "decode_numpy_dict_values" to convert numpy to lists using tolist.

Is this feature requested elsewhere, or was it intentionally not considered as a use-case? If not, happy to contribute code.

[shoyer]

We could add a keyword argument to control this, e.g., numpy=True to indicate that data should be kept as NumPy arrays.

[nicain]

Yeah, my thoughts exactly. I am almost done with this in my fork... numpy might be confusing for those who will see the module name when they look at it. How does a kwarg:

def to_dict(self, tolist=True):

sound?

[shoyer]

I'll let others chime in on the API design here but yes that sounds pretty reasonable, too.

[chrisbarber]

Could have a callable serializer kwarg that defaults to np.ndarray.tolist. I have a use case where I would pass in np.ndarray.tobytes for this. But then again, I could just use numpy=True or tolist=False and then walk the dict myself.

[nicain]

@chrisbarber Maybe a method to_byte_dict that (in implementation) takes the output from numpy=True and walks the dict with tobytes?

[chrisbarber]

@nicain, for sure. Probably best for the API's sake to stick to the simplicity of a flag.

[nicain]

@chrisbarber

Here is my example implementation: https://github.com/nicain/xarray/blob/fa86e3d38ebf4e641cafd963a5b69a77539b931d/xarray/core/dataarray.py#L1388

@shoyer Can you point me to where to edit documentation and unit test best practices?

[shoyer]

We need to write a contributing guide!

But in brief:

• We use the NumPy docstring convention: https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt
• For unit tests, we use pytest, with helper functions/methods for verifying comparisons between objects. Take a look at the existing tests for examples to emulate:

  xarray/xarray/tests/test_dataset.py

  Line 2402 in 25d1855

  def test_to_and_from_dict(self):

See also: https://github.com/pydata/xarray/pull/1485/files

[nicain]

Sweet, this is exactly what I am used to! Ill give a shot to writing a few tests later today, and make a PR. Can you suggest some core devs I can ping for a PR review? Ill ask @chrisbarber (we work together) to review, but I'd like more eyeballs on it.

[shoyer]

@nicain myself, @jhamman and/or @MaximilianR are good choices for reviewers, thanks!

[stale]

In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity

If this issue remains relevant, please comment here or remove the stale label; otherwise it will be marked as closed automatically

[jmccreight]

I'd be interested in reviving this, this is exactly what I want to achieve. It's not clear if there was some reason this never went ahead. I looked around but didnt find anything. LMK if it there's some reason not to pursue it. THanks

[jhamman]

@jmccreight - I don't think there is any specific reason this didn't get done. Still open to contributions here if you are interested.

[jmccreight]

The PR #7739 is available for review. @jhamman @dcherian would be my choices. i think this is pretty straight forward. I suppose the name of the kwarg being numpy_data is debatable. I based this on the discussion of numpy vs tolist above, preferring the former but acknowledging the comment that a package name as an arg is odd. Could do as_numpy or something slightly different.

[dcherian]

Perhaps we should have array_to_list: bool instead. If False, we just preserve the underlying array type.

Then the user could do ds.as_numpy().to_dict(array_to_list=False) to always get numpy arrays as #7739