You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Parquet spec prescribes that for backward compatibility the legacy converted type TIMESTAMP_MILLIS should be interpreted as logical type TimestampType (isAdjustedToUTC = true, unit = MILLIS), but that's currently not the case.
To Reproduce
Read a modern Parquet file with schema:
message x {
REQUIRED INT64 system_time (TIMESTAMP(MILLIS,true));
}
observe data type Timestamp(Millisecond, Some("UTC"))
Read an old Parquet file with schema:
message x {
REQUIRED INT64 system_time (TIMESTAMP_MILLIS);
}
observe data type: Timestamp(Millisecond, None)
TIMESTAMP_MILLIS gets incorrectly interpreted as naive/local timestamp.
Expected behavior
Because spec prescribes that TIMESTAMP_MILLIS should be interpreted as TIMESTAMP(MILLIS,true) I expected to see a timestamp with UTC timezone.
Describe the bug
Parquet spec prescribes that for backward compatibility the legacy converted type
TIMESTAMP_MILLIS
should be interpreted as logical typeTimestampType (isAdjustedToUTC = true, unit = MILLIS)
, but that's currently not the case.To Reproduce
Read a modern Parquet file with schema:
observe data type
Timestamp(Millisecond, Some("UTC"))
Read an old Parquet file with schema:
observe data type:
Timestamp(Millisecond, None)
TIMESTAMP_MILLIS
gets incorrectly interpreted as naive/local timestamp.Expected behavior
Because spec prescribes that
TIMESTAMP_MILLIS
should be interpreted asTIMESTAMP(MILLIS,true)
I expected to see a timestamp with UTC timezone.Additional context
These lines seem to be the source of the problem
The text was updated successfully, but these errors were encountered: