Any way to change datetime64 resolution?

[altaurog]

xref to #7307

Converting from datetime64[ns] to datetime64[m] is disallowed, as mentioned in issue #3416, apparently because of a numpy bug. It looks to me like numpy does this correctly, however. It would be nice if this could work without resorting to raw numpy arrays:

In [89]: s = pd.Series(['20140330','19740821'])
In [90]: pd.to_datetime(s)
Out[90]: 
0   2014-03-30
1   1974-08-21
dtype: datetime64[ns]
In [91]: pd.to_datetime(s).values
Out[91]: 
array(['2014-03-30T03:00:00.000000000+0300',
       '1974-08-21T03:00:00.000000000+0300'], dtype='datetime64[ns]')
In [92]: pd.to_datetime(s).values.astype('datetime64[h]')
Out[92]: array(['2014-03-30T03+0300', '1974-08-21T03+0300'], dtype='datetime64[h]')

My other workaround, which is slightly less clumsy, is do just convert manually, since casting to int64 is permitted:

In [103]: pd.to_datetime(s).astype('int64').div(6e10) # minutes since epoch

But it'd be nicer not to have to do all that.

In [93]: pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.3.final.0
python-bits: 64
OS: Linux
OS-release: 3.2.0-4-amd64
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.13.1
Cython: 0.20
numpy: 1.8.0
scipy: None
statsmodels: None
IPython: 1.2.1
sphinx: 1.1.3
patsy: None
scikits.timeseries: None
dateutil: 1.5
pytz: 2012c
bottleneck: 0.8.0
tables: 3.1.0
numexpr: 2.3.1
matplotlib: 1.3.1
openpyxl: None
xlrd: None
xlwt: 0.7.4
xlsxwriter: None
sqlalchemy: None
lxml: None
bs4: None
html5lib: 0.999
bq: None
apiclient: None

[jreback]

Well you are actually looking for this.

In [31]: pd.to_datetime(s).astype('int64')/np.timedelta64(1,'m').astype('timedelta64[ns]').astype('int64')
Out[31]: 
0    23268960
1     2437920
dtype: float64

We don't allow direct conversions because its simply too complicated to keep anything other than datetime64[ns] internally (nor necessary at all). It could be done, but not very useful IMHO.

This is currently NotImplemented, but is very straightforward to do.

In [32]: pd.to_datetime(s)/np.timedelta64(1,'m')

TypeError: can only operate on a datetime with a rhs of a timedelta for addition and subtraction,  but the operator [__truediv__] was passed

Could also support pd.to_datetime(s).astype('timedelta64[m]') in the same way

I'll mark this as an enhancement

Want to submit a PR?

[altaurog]

Could also support pd.to_datetime(s).astype('timedelta64[m]')

Yes! That's what I'm looking for. But what would the result be, if you want to keep everything in timedelta64[ns] internally?

I understand the motivation for keeping the code simple. Personally, I am really sold on numpy's support for timestamps with varying precision.

[jreback]

result would be the same as above
a float64 (or int64) dtyped series

only thing that makes sense

[jreback]

what are u using this for?

[altaurog]

I'm loading data on/off an embedded device and also serving visualizations of said data. I'm using minutes since epoch as my time representation since that is a reasonable precision for me and fits into a 32-bit int.

[altaurog]

BTW I am impressed not only with the speed of pandas, but also by the speed with which you respond to issues here.

[jreback]

thanks

to_json can do thing like this FYI
(but may not help in your use case)

[cancan101]

See also: #4338

[michaelaye]

is this (i.e. internally all is in nanoseconds) the reason why a timedelta[s] does not display correctly?
https://www.dropbox.com/s/spbn32fdoyqfn4i/Screenshot%202014-09-08%2021.54.57.pdf?dl=0

[jreback]

@michaelaye #8184 will convert this properly

[jreback]

this is essentially covered by #7307

so going to close this in favor of that if internal resolution is changed (then this will be possible)