ValueError: Buffer has wrong number of dimensions (expected 1, got 2)

[naught101]

grab a copy of the file http://nh.id.au/data/ocean_vort.nc.gz, and gunzip it. It's a file with some ocean vorticity fields, from the MOM4 model. The ncdump -h ocean_vort.nc results don't look too odd to me.

If I run:

import xray

ds = xray.open_dataset('ocean_vort.nc')
ds

I get the following error:

ValueError                                Traceback (most recent call last)
/data/downloads/software/ipython/IPython/core/formatters.py in __call__(self, obj)
    695                 type_pprinters=self.type_printers,
    696                 deferred_pprinters=self.deferred_printers)
--> 697             printer.pretty(obj)
    698             printer.flush()
    699             return stream.getvalue()

/data/downloads/software/ipython/IPython/lib/pretty.py in pretty(self, obj)
    382                             if callable(meth):
    383                                 return meth(obj, self, cycle)
--> 384             return _default_pprint(obj, self, cycle)
    385         finally:
    386             self.end_group()

/data/downloads/software/ipython/IPython/lib/pretty.py in _default_pprint(obj, p, cycle)
    502     if _safe_getattr(klass, '__repr__', None) not in _baseclass_reprs:
    503         # A user-provided repr. Find newlines and replace them with p.break_()
--> 504         _repr_pprint(obj, p, cycle)
    505         return
    506     p.begin_group(1, '<')

/data/downloads/software/ipython/IPython/lib/pretty.py in _repr_pprint(obj, p, cycle)
    700     """A pprint that just redirects to the normal repr function."""
    701     # Find newlines and replace them with p.break_()
--> 702     output = repr(obj)
    703     for idx,output_line in enumerate(output.splitlines()):
    704         if idx:

/home/naught101/miniconda3/envs/science/lib/python3.4/site-packages/xray/core/dataset.py in __repr__(self)
    885 
    886     def __repr__(self):
--> 887         return formatting.dataset_repr(self)
    888 
    889     @property

/home/naught101/miniconda3/envs/science/lib/python3.4/site-packages/xray/core/formatting.py in dataset_repr(ds)
    271 
    272     summary.append(coords_repr(ds.coords, col_width=col_width))
--> 273     summary.append(vars_repr(ds.data_vars, col_width=col_width))
    274     if ds.attrs:
    275         summary.append(attrs_repr(ds.attrs))

/home/naught101/miniconda3/envs/science/lib/python3.4/site-packages/xray/core/formatting.py in _mapping_repr(mapping, title, summarizer, col_width)
    208     summary = ['%s:' % title]
    209     if mapping:
--> 210         summary += [summarizer(k, v, col_width) for k, v in mapping.items()]
    211     else:
    212         summary += [EMPTY_REPR]

/home/naught101/miniconda3/envs/science/lib/python3.4/site-packages/xray/core/formatting.py in <listcomp>(.0)
    208     summary = ['%s:' % title]
    209     if mapping:
--> 210         summary += [summarizer(k, v, col_width) for k, v in mapping.items()]
    211     else:
    212         summary += [EMPTY_REPR]

/home/naught101/miniconda3/envs/science/lib/python3.4/site-packages/xray/core/formatting.py in summarize_var(name, var, col_width)
    172 def summarize_var(name, var, col_width):
    173     show_values = _not_remote(var)
--> 174     return _summarize_var_or_coord(name, var, col_width, show_values)
    175 
    176 

/home/naught101/miniconda3/envs/science/lib/python3.4/site-packages/xray/core/formatting.py in _summarize_var_or_coord(name, var, col_width, show_values, marker, max_width)
    154     front_str = first_col + dims_str + ('%s ' % var.dtype)
    155     if show_values:
--> 156         values_str = format_array_flat(var, max_width - len(front_str))
    157     else:
    158         values_str = '...'

/home/naught101/miniconda3/envs/science/lib/python3.4/site-packages/xray/core/formatting.py in format_array_flat(items_ndarray, max_width)
    130     # print at least one item
    131     max_possibly_relevant = max(int(np.ceil(max_width / 2.0)), 1)
--> 132     relevant_items = first_n_items(items_ndarray, max_possibly_relevant)
    133     pprint_items = format_items(relevant_items)
    134 

/home/naught101/miniconda3/envs/science/lib/python3.4/site-packages/xray/core/formatting.py in first_n_items(x, n_desired)
     54         indexer = _get_indexer_at_least_n_items(x.shape, n_desired)
     55         x = x[indexer]
---> 56     return np.asarray(x).flat[:n_desired]
     57 
     58 

/home/naught101/miniconda3/envs/science/lib/python3.4/site-packages/numpy/core/numeric.py in asarray(a, dtype, order)
    472 
    473     """
--> 474     return array(a, dtype, copy=False, order=order)
    475 
    476 def asanyarray(a, dtype=None, order=None):

/home/naught101/miniconda3/envs/science/lib/python3.4/site-packages/xray/core/common.py in __array__(self, dtype)
     73 
     74     def __array__(self, dtype=None):
---> 75         return np.asarray(self.values, dtype=dtype)
     76 
     77     def __repr__(self):

/home/naught101/miniconda3/envs/science/lib/python3.4/site-packages/xray/core/dataarray.py in values(self)
    332     def values(self):
    333         """The array's data as a numpy.ndarray"""
--> 334         return self.variable.values
    335 
    336     @values.setter

/home/naught101/miniconda3/envs/science/lib/python3.4/site-packages/xray/core/variable.py in values(self)
    269     def values(self):
    270         """The variable's data as a numpy.ndarray"""
--> 271         return _as_array_or_item(self._data_cached())
    272 
    273     @values.setter

/home/naught101/miniconda3/envs/science/lib/python3.4/site-packages/xray/core/variable.py in _data_cached(self)
    235     def _data_cached(self):
    236         if not isinstance(self._data, np.ndarray):
--> 237             self._data = np.asarray(self._data)
    238         return self._data
    239 

/home/naught101/miniconda3/envs/science/lib/python3.4/site-packages/numpy/core/numeric.py in asarray(a, dtype, order)
    472 
    473     """
--> 474     return array(a, dtype, copy=False, order=order)
    475 
    476 def asanyarray(a, dtype=None, order=None):

/home/naught101/miniconda3/envs/science/lib/python3.4/site-packages/xray/core/indexing.py in __array__(self, dtype)
    292     def __array__(self, dtype=None):
    293         array = orthogonally_indexable(self.array)
--> 294         return np.asarray(array[self.key], dtype=None)
    295 
    296     def __getitem__(self, key):

/home/naught101/miniconda3/envs/science/lib/python3.4/site-packages/xray/conventions.py in __getitem__(self, key)
    416 
    417     def __getitem__(self, key):
--> 418         return decode_cf_timedelta(self.array[key], units=self.units)
    419 
    420 

/home/naught101/miniconda3/envs/science/lib/python3.4/site-packages/xray/conventions.py in decode_cf_timedelta(num_timedeltas, units)
    166     num_timedeltas = _asarray_or_scalar(num_timedeltas)
    167     units = _netcdf_to_numpy_timeunit(units)
--> 168     result = pd.to_timedelta(num_timedeltas, unit=units, box=False)
    169     # NaT is returned unboxed with wrong units; this should be fixed in pandas
    170     if result.dtype != 'timedelta64[ns]':

/home/naught101/miniconda3/envs/science/lib/python3.4/site-packages/pandas/util/decorators.py in wrapper(*args, **kwargs)
     87                 else:
     88                     kwargs[new_arg_name] = new_arg_value
---> 89             return func(*args, **kwargs)
     90         return wrapper
     91     return _deprecate_kwarg

/home/naught101/miniconda3/envs/science/lib/python3.4/site-packages/pandas/tseries/timedeltas.py in to_timedelta(arg, unit, box, errors, coerce)
     64         return _convert_listlike(arg, box=box, unit=unit, name=arg.name)
     65     elif is_list_like(arg):
---> 66         return _convert_listlike(arg, box=box, unit=unit)
     67 
     68     # ...so it must be a scalar value. Return scalar.

/home/naught101/miniconda3/envs/science/lib/python3.4/site-packages/pandas/tseries/timedeltas.py in _convert_listlike(arg, box, unit, name)
     47             value = arg.astype('timedelta64[{0}]'.format(unit)).astype('timedelta64[ns]', copy=False)
     48         else:
---> 49             value = tslib.array_to_timedelta64(_ensure_object(arg), unit=unit, errors=errors)
     50             value = value.astype('timedelta64[ns]', copy=False)
     51 

pandas/tslib.pyx in pandas.tslib.array_to_timedelta64 (pandas/tslib.c:47353)()

ValueError: Buffer has wrong number of dimensions (expected 1, got 2)

Any idea what might be causing that problem?

[naught101]

xray.version.version == '0.6.1' from anaconda, on python 3.4

[shoyer]

What version of netcdf4-Python are you using? I have a strong hunch this is the same issue as #662

[naught101]

$ conda list |grep -i cdf
libnetcdf                 4.3.3.1                       1  
netcdf4                   1.1.9               np110py34_0  

[naught101]

I updated to 1.2.1 from pypi, it still fails.

[shoyer]

Did you try specifying a different engine to read the netcdf file? That might at least give you different error ;).

On Mon, Nov 23, 2015 at 9:31 PM, naught101 [email protected]
wrote:

I updated to 1.2.1 from pypi, it still fails.

Reply to this email directly or view it on GitHub:
#665 (comment)

[naught101]

Same problem with scipy - I'm not sure what I need to install to get pydap and h5netcdf working...

[naught101]

@shoyer: the file is publically available, have you tried checking it on your set-up? The file is about 35Mb..

[shoyer]

OK, I just tried this out and this is a legit bug.

The array that needs to be converted to time deltas is 2D, and pandas can't handle that. We need to flatten out the array in decode_cf_timedelta and then unflatten it, similarly to what we do in decode_cf_datetime.

Any interest in putting together a pull request with a fix?

[naught101]

Sorry, this is just something I stumbled across in a workshop, it's not a problem that's affecting my work, so I don't have much time for it. But I'll make the MOM developers aware of it - they might have an interest in helping out.

[shoyer]

No worries -- thanks for the bug report! I'm sure someone will get to this
eventually.

On Tue, Nov 24, 2015 at 2:15 PM, naught101 [email protected] wrote:

Sorry, this is just something I stumbled across in a workshop, it's not a
problem that's affecting my work, so I don't have much time for it. But
I'll make the MOM developers aware of it - they might have an interest in
helping out.

—
Reply to this email directly or view it on GitHub
#665 (comment).

[spencerahill]

@spencerkclark and I have come across the same problem in one of the atmospheric models we work with, but in our case in addition to the time bounds array the latitude and longitude bounds are also 2D. I mention this because I suspect this wouldn't be resolved by @shoyer 's previous comment re: fixing decode_cf_timedelta. I was wrong here; see comments in #667

[shoyer]

@spencerahill if you get a similar error with 2d latitude/longitude arrays please do report it in a separate issue. I'm pretty sure this one is specific to timedeltas.

[spencerahill]

@shoyer Sure, just did: #667. My suspicion is that it's ultimately the same underlying issue, which is simply that these bound arrays have a 2nd dimension that xray is having a hard time dealing with. Nevertheless, that it occurs in non-time coordinates as well implies that dealing with it solely through time-related methods won't solve everything. I was wrong here; see comments in #667.

[spencerkclark]

@spencerahill I recall we only had this specific issue with the 2D time_bounds variable. Perhaps you have a specific example in mind, but in my experience xray handles reading in 2D lat and lon bounds variables OK.