-
Notifications
You must be signed in to change notification settings - Fork 784
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
datetime: support timezone bindings #1588
Conversation
30d35ba
to
bd6ecde
Compare
@@ -279,7 +252,7 @@ impl PyTime { | |||
minute: u8, | |||
second: u8, | |||
microsecond: u32, | |||
tzinfo: Option<&PyObject>, | |||
tzinfo: Option<&PyTzInfo>, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldn't this be a breaking change? It may probably affect pyo3-chrono crate. Maybe need to ping the author if this is indeed the case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, and we'll release this as part of the 0.14 release. I've notified this in the CHANGELOG, and we are still pre-1.0, after all. If it turns out to be a big problem we can revert it later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can't quite remember why I went with &PyObject
instead of &PyTzInfo
here originally. It's entirely possible that I misunderstood how inheritance in Rust and Python can play together well, The datetime.tzinfo
class is an abstract base class (but I think one that was created before Python supported either abstract base classes or Protocols).
In theory it should be a trait, but in practice datetime
enforces that it is an actual subclass of datetime.tzinfo
and I don't know if I'll be able to change that any time soon. Presumably it is safe to change this from PyTzInfo
back to PyObject
after the 1.0 release, since all PyTzInfo
objects are PyObject
s but not vice versa?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, I remember why it has to be &PyObject
now — tzinfo
can also be None
(and in fact probably is None
in the common case, since datetime.timezone
is not a very good time zone representation but datetimes are useful anyway).
I'm actually fairly surprised that this doesn't break any tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, sorry to keep going back and forth here, but I forgot that this is Option<&PyTzInfo>
, and that the None
is handled by the None
case of Option
, so I'm back to not remembering why this was PyObject
instead of PyTzInfo
.
src/ffi/datetime.rs
Outdated
@@ -445,12 +445,11 @@ pub static PyDateTimeAPI: _PyDateTimeAPI_impl = _PyDateTimeAPI_impl { | |||
inner: GILOnceCell::new(), | |||
}; | |||
|
|||
/// Safe wrapper around the Python C-API global `PyDateTime_TimeZone_UTC`. This follows a similar | |||
/// Wrapper around the Python C-API global `PyDateTime_TimeZone_UTC`. This follows a similar |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this not a safe wrapper now?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, I made it less safe because it gives you &'static *mut ffi::PyObject
, which is a pointer and thus less safe to use. The safe wrapper is now the pyo3::types::timezone_utc()
function.
src/types/datetime.rs
Outdated
/// Equivalent to `datetime.timezone.utc` | ||
#[cfg(all(Py_3_7, not(PyPy)))] | ||
pub fn timezone_utc(py: Python) -> &PyTzInfo { | ||
unsafe { | ||
&*(&*ffi::PyDateTime_TimeZone_UTC as *const *mut ffi::PyObject | ||
as *const crate::Py<PyTzInfo>) | ||
} | ||
.as_ref(py) | ||
} | ||
|
||
/// Equivalent to `datetime.timezone(offset)` | ||
#[cfg(all(Py_3_7, not(PyPy)))] | ||
pub fn timezone_from_offset<'py>(py: Python<'py>, offset: &PyDelta) -> PyResult<&'py PyTzInfo> { | ||
unsafe { py.from_owned_ptr_or_err(ffi::PyTimeZone_FromOffset(offset.as_ptr())) } | ||
} | ||
|
||
/// Equivalent to `datetime.timezone(offset, name)` | ||
#[cfg(all(Py_3_7, not(PyPy)))] | ||
pub fn timezone_from_offset_and_name<'py>( | ||
py: Python<'py>, | ||
offset: &PyDelta, | ||
name: &str, | ||
) -> PyResult<&'py PyTzInfo> { | ||
let name = name.into_py(py); | ||
unsafe { | ||
py.from_owned_ptr_or_err(ffi::PyTimeZone_FromOffsetAndName( | ||
offset.as_ptr(), | ||
name.as_ptr(), | ||
)) | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldn't these be easier to discover if we put it within impl PyTzInfo
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I went back and forth on that. Yes, it probably would. However I wasn't sure if that was totally "correct" because they're actually creating a subtype. If you also think it'd be nicer to have them on impl PyTzInfo
then I'm persuaded that's the right thing to do 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, I am just thinking that datetime
module could have tons of functions already and separating these would increase those later on, since these will only create PyTzInfo
I don't see why not put it there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I definitely don't think this should go in impl PyTzInfo
. PyTzInfo
exists only to be a base class, so you shouldn't really be able to get instances of it. Despite Rust's lack of nominal type hierarchies, it needs Rust wrappers so you can do stuff like this:
pyo3/examples/pyo3-pytests/src/datetime.rs
Lines 181 to 202 in 3ee84a3
#[pyclass(extends=PyTzInfo)] | |
pub struct TzClass {} | |
#[pymethods] | |
impl TzClass { | |
#[new] | |
fn new() -> Self { | |
TzClass {} | |
} | |
fn utcoffset<'p>(&self, py: Python<'p>, _dt: &PyDateTime) -> PyResult<&'p PyDelta> { | |
PyDelta::new(py, 0, 3600, 0, true) | |
} | |
fn tzname(&self, _py: Python<'_>, _dt: &PyDateTime) -> String { | |
String::from("+01:00") | |
} | |
fn dst(&self, _py: Python<'_>, _dt: &PyDateTime) -> Option<&PyDelta> { | |
None | |
} | |
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't mean with &self
, I just mean something like
impl PyTzinfo {
fn utcoffset<'py>(py: Python<'py>, offset: &PyDelta) -> PyResult<&'py PyTzInfo> {
...
}
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pickfire Regardless, I don't think it's the right abstraction, because it implies that this is a constructor for PyTzInfo
, which is an abstract base class.
I do think it would make sense for these methods to turn into a wrapper class for datetime.timezone
like the others (minus the _Check
methods and such), in which case these could be alternate constructors for PyFixedTimezone
or whatever it's called.
I'm not sure I understand what is complicated about having a new type representing All of the other type wrappers we have are Rust types with constructors corresponding to the equivalent C API constructors. |
But this one is a python wrapper. Wouldn't |
We haven't yet had to model the inheritance hierarchy of types like this. Would ... I'm going to work on a couple of PRs to see if implementing |
It has unfortunately been a while since I had a halfway decent understanding of the way Python inheritance is modeled in PyO3 (if I ever did). This sounds appealing to me, but is there a more general problem that sometimes your Is it not sufficient to implement In this case, it would be really nice if we could use a trait instead, since the actual memory layout of the |
One other comment on this: Other than exposing the Plus, it's generally not a good idea to use |
Yes, and if I implemented this we'd have
Something like this could be done, yes. However I'm a bit uneasy about adding new trait mappings like I think that changes like #1056 need to be resolved before we commit to more ways to interconvert Python types. I'm similarly uneasy about the
Understood, however I think there's also a practical notion that these APIs would be helpful for chrono as current, and these are exposed in the Python C-API so it's reasonable for PyO3 to expose these too. Overall, I'm leaning towards merging this PR as-is, because it's a simple and practical approach even if the new functions aren't super discoverable. Perhaps their lack of discoverability may be a feature 😉 We can make future improvements to this API as PyO3 and the Python C-API evolve... |
I'm still kinda -1 for the same reason that I didn't like the idea of adding In general, it's not going to be useful to implement specifically impl ToPyObject for FixedOffset {
fn to_object(&self, py: pyo3::Python) -> pyo3::Result<pyo3::PyObject> {
let dt_module = py.import("datetime")?;
let seconds_offset = self.utc_minus_local(); // Throw exception if not ±24h?
let td = pyo3::PyDelta::new(py, seconds_offset, 0, true)?;
dt_module.call1("timezone", (td,))?
}
} This would be slightly slower than calling the C API directly, but if you are concerned about that, a cache would likely make it much faster than even calling the C API directly; since there are a maximum of ~172,800 possible fixed offsets, and in practice there's maybe 30-40 that you'll actually encounter, any process doing a large number of conversions of fixed offsets to So at the end of the day, I'd say that we should concentrate on creating nice, consistent bindings for CPython in |
I think you missed the context here. What I am trying to do is provide chrono and pyo3 integration in which the python timezone is almost the same as chrono provided functions. chrono also does not provide timezone, so rather than having users do the conversion, we can just support it on library level so the experience is seamless.
I think that's probably not a good idea in the long run, keeping it align with the C API might be a better idea.
Other than chrono there are also other time libraries or other stuff that needs the timezone. I think implementing it here is better than having everyone have their own implementation which they can easily get wrong (like me, I still can't figure out how to do the FFI thing by myself). The code you showed doesn't seemed like a good idea. pyo3 is exposing safe wrappers over python API. Of course not all parts of C API is necessary but doing it this way seemed weird, the |
The problem is that
The fn timezone_from_offset<'py>(py: Python<'py>, offset: &PyDelta) -> PyResult<&'py PyTzInfo> {
let dt_mod = py.import("datetime")?;
dt_mod.call1("timezone", (offset))?
} This is a trivial operation and it should be simple to do it from within your Additionally, it returns the exact same object that you'd get if you were using the |
I still don't see what is the issue with fixed offsets? Can you please explain it or open an issue with chrono? Ah, I didn't know the code will work. Will try it out later and if this patch didn't get merged I will do that. But if other people come do they need to go search the pull request and find that comment? I think having a function there is simpler such that others does not have to even worry about the low level stuff to have to go search the issues and pull requests and read the source code like I did to figure out how to do the same thing but using a lower-level API, just using a high-level API like the one in python would be easier to find and figure out how it can be used. |
Python offsets can have sub-second precision as well, as of Python 3.7 or 3.8 (I think 3.7). It is also the case that in general, Python datetimes should have a The biggest problem I have with fixed offsets is that they are rarely actually all that useful. Generally speaking, you either want an absolute time, which represents a fixed point in time (e.g. UTC, an epoch timestamp, etc) or you want a civil time, which represents what the clock will say on the wall at a specific time in a specific place. Absolute times are usually useful for recording when something happened in the past (e.g. timestamps on logs) and civil times are usually useful for human-centric events that may take place in the future. When working with absolute times, it's almost always sufficient to record it and work with it in UTC and convert it to local civil time when you want to display it to a human. When working with future times it is not appropriate to use UTC or a fixed offset thereof — imagine we plan an event at 14:00 a month from now, and then the country we're meeting in decides to eliminate DST. If we had recorded the meeting as 14:00-04:00 — in local time that is now 13:00, even though from our perspective we always want the meeting to be at 14:00. The way Python works is that you attach a
First of all, I don't think my suggestion here was terribly complicated. If you do any appreciable amount of coding with Rust ↔ Python or C ↔ Python interop, you'll quickly find that there aren't C API hooks for even most of the standard library, so if you can't find a way to accomplish something using the C API, you just make a Python function call and get a I am in a weird position here because I do think that we should expose |
If it's a bug that people rely on. It's not a bug. It's a feature. -- Linus At least in this case, based on what you said taking DST into account it is a bad design. But considering that RFC 3339 and ISO 8601 supports fixed offsets by default in datetime but not IANA or something similar, I think it is still good to have an API for it, at least python already does it even though it is not recommended, it is not really a foot gun for this to be exact, even chrono does the same thing. But yeah, in the docs we can just put a note that it is more recommended to use IANA rather than the fixed offset (but we don't have a pointer to say what to use yet). I think it still depends on @davidhewitt call. If he merged then I will use the convenience functions, if he didn't then I can just use the one you showed. |
Some time has passed, is there any decisions on this or what's pending? |
Overall I'm still leaning towards merging this eventually. I understand the concerns that this is not IANA timezone, however we also don't have a C API for the The main reason I have not been rushing to merge this is that it's currently a bit clunky to use custom |
Okay, then I guess I will just use the other method to call python code directly for chrono, when I have the time to work on that. |
The concern is less that this isn't an IANA timezone and more that these shouldn't be useful. I agree that we should probably expose the relevant part of the C API just for completeness, but I don't agree with adding convenience functions for constructing these. Giving people low-friction options for doing things they shouldn't do is a recipe for wide adoption of bad practices. One reason it seems unlikely that we would bother with |
That's fine. I hope that one day we'll have easy ways to generate type-safe wrappers for Python types to reduce the amount of dynamic code users have to write. We could eventually use that to wrap I started such a project at https://github.com/davidhewitt/pyo3-pytypes, however I think it'll be a long time before I next get a chance to sit down and try again to make it happen. |
I think we should also have impl PyTzInfoAccess for PyDateTime {
fn get_tzinfo(&self) -> Option<PyObject> {
let ptr = unsafe { PyDateTime_DATE_GET_TZINFO(self.as_ptr()) };
if unsafe { ffi::Py_None() } == ptr {
None
} else {
Some(ptr as PyObject) // this
}
}
} |
I'd suggest opening as a separate PR, it's probably going to be quite a while before I have a chance to finish this off unfortunately. |
d1edb34
to
0cdf99b
Compare
After a long delay, I'm leaning towards merging this for 0.17. I appreciate the concern that this does not expose IANA timezones. While that is perhaps something that we can consider in future, for now the Python C API lacks any bindings for |
As I said before, this not exposing IANA time zones is not the problem. The problem is that these fixed offsets are both not useful and they seem like they are useful, which is a dangerous combination. |
@pganssle Would it resolve your concern if I took the two |
Yes, I think those two are the main problem I have with this change, the other stuff all seems positive. The only comment I'd have about |
0cdf99b
to
21af848
Compare
21af848
to
7babd13
Compare
The complication for safety on the Rust side is that if the singleton hasn't yet been imported then we need to hold the GIL to safely import. This could be done behind the scenes in a Thanks for all the reviews, proceeding to merge this. |
{ | ||
let dt = PyDateTime::from_timestamp(py, 100.0, Some(timezone_utc(py))).unwrap(); | ||
py_run!( | ||
py, | ||
dt, | ||
"import datetime; assert dt == datetime.datetime.fromtimestamp(100, datetime.timezone.utc)" | ||
); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think your indentation here is off.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, #2508 to fix
This PR implements safe wrappers for the Python 3.7 timezone bindings described in #207.
Rather than try to do anything complicated with a new type representing exactly
datetime.timezone
, I return&PyTzInfo
from the three new functionstimezone_utc()
,timezone_from_offset()
andtimezone_from_offset_and_name()
. I think this is good enough to be useful;datetime.timezone
is a subclass ofdatetime.tzinfo
so doing this is correct.I also adjusted a few constructors taking
tzinfo
arguments to takeOption<&PyTzInfo>
instead ofOption<&PyObject>
.Finally, I tweaked the ffi symbol
PyDateTime_TimeZone_UTC
I added recently in #1573 to resolve to&'static *mut ffi::PyObject
, which I think is a more correct representation of the "unsafe ffi" nature of that global.Closes #207
cc @pganssle in case you might be interested / have an opinion on this.