datetime serialization/deserialization broken

[shyamd]

datetime objects with time zone info break the serializatin because __str__ doesn't output in the format that monty expects. Example:
datetime.datetime(2018, 12, 19, 21, 45, 41, 99000, tzinfo=datetime.timezone.utc).__str__()
yields
"2018-12-19 21:45:41.099000+00:00"

When deserializing, this causes monty to throw an error because the two datetime formatting strings it tries are:

• "%Y-%m-%d %H:%M:%S.%f"
• "%Y-%m-%d %H:%M:%S"

resulting in value error like this from within strptime:
ValueError: unconverted data remains: .967000+00:00

We could just add the timezone format, or just use a formatting string when outputting from datetime to a string during serialization. There are also generic datetime parsing libraries like dateutil

[DanielYang59]

I could confirm this issue for strptime with time zone info, materialsproject/pymatgen#3705 (comment).

import datetime

start_time_old = datetime.datetime.utcnow()  # 2024-03-25 08:46:23.748342 

start_time_new = datetime.datetime.now(datetime.UTC)  # 2024-03-25 08:46:23.748472+00:00

print(start_time_old, start_time_new)  


dt = datetime.datetime.strptime(
    str(start_time_old), "%Y-%m-%d %H:%M:%S.%f"
)
print(dt)


dt = datetime.datetime.strptime(
    str(start_time_new), "%Y-%m-%d %H:%M:%S.%f"
)
print(dt)

Which raises:

Traceback (most recent call last):
  File "/Users/yang/developer/test/test.py", line 16, in <module>
    dt = datetime.datetime.strptime(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/_strptime.py", line 554, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/_strptime.py", line 336, in _strptime
    raise ValueError("unconverted data remains: %s" %
ValueError: unconverted data remains: +00:00

[shyuep]

I don't have time to fix this. But feel free to submit a PR.

[DanielYang59]

I'm currently working on this, here I enclose a code snippet to recreate this issue from monty's side (for my own record):

import json
import datetime
from monty.json import MontyEncoder, MontyDecoder

# created_at = datetime.datetime.utcnow()   # deprecated API
created_at = datetime.datetime.now(tz=datetime.timezone.utc)

data = json.loads(json.dumps(created_at, cls=MontyEncoder))

created_at = MontyDecoder().process_decoded(data)

Gives:

Traceback (most recent call last):
  File "/home/yang/monty/monty/json.py", line 750, in process_decoded
    dt = datetime.datetime.strptime(
  File "/usr/lib/python3.10/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/usr/lib/python3.10/_strptime.py", line 352, in _strptime
    raise ValueError("unconverted data remains: %s" %
ValueError: unconverted data remains: +00:00

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/yang/monty/recreate.py", line 10, in <module>
    created_at = MontyDecoder().process_decoded(data)
  File "/home/yang/monty/monty/json.py", line 754, in process_decoded
    dt = datetime.datetime.strptime(
  File "/usr/lib/python3.10/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/usr/lib/python3.10/_strptime.py", line 352, in _strptime
    raise ValueError("unconverted data remains: %s" %
ValueError: unconverted data remains: .914911+00:00