Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Enable SPARK-10634 timestamp test case #5090

Merged
merged 3 commits into from
Mar 27, 2024
Merged

[VL] Enable SPARK-10634 timestamp test case #5090

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0f3e72e
fix repo for test
liujiayi771 Mar 23, 2024
b69e9aa
Remove exlcude for SPARK-10634 timestamp written and read as INT64 - …
liujiayi771 Mar 23, 2024
0ef492d
Revert "fix repo for test"
liujiayi771 Mar 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions gluten-ut/spark32/src/test/scala/io/glutenproject/utils/velox/VeloxTestSettings.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -857,7 +857,6 @@ class VeloxTestSettings extends BackendTestSettings {
// decimal failed ut
.exclude("SPARK-34212 Parquet should read decimals correctly")
// Timestamp is read as INT96.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This message was added when int64 type was unsupported for timestamp reader. Could you update the message to illustrate why below tests are still disabled?

.exclude("Migration from INT96 to TIMESTAMP_MICROS timestamp type")
.exclude("SPARK-10365 timestamp written and read as INT64 - TIMESTAMP_MICROS")

Copy link
Contributor Author

@liujiayi771 liujiayi771 Mar 25, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two unit tests are still failing, but it's not the same issue. We can continue investigating to find the root cause.

.exclude("SPARK-10634 timestamp written and read as INT64 - truncation")
.exclude("Migration from INT96 to TIMESTAMP_MICROS timestamp type")
.exclude("SPARK-10365 timestamp written and read as INT64 - TIMESTAMP_MICROS")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you tried above two unit tests, and are they related to the same issue?

// Rewrite because the filter after datasource is not needed.
Expand All @@ -869,7 +868,6 @@ class VeloxTestSettings extends BackendTestSettings {
// decimal failed ut
.exclude("SPARK-34212 Parquet should read decimals correctly")
// Timestamp is read as INT96.
.exclude("SPARK-10634 timestamp written and read as INT64 - truncation")
.exclude("Migration from INT96 to TIMESTAMP_MICROS timestamp type")
.exclude("SPARK-10365 timestamp written and read as INT64 - TIMESTAMP_MICROS")
// Rewrite because the filter after datasource is not needed.
Expand Down
2 changes: 0 additions & 2 deletions gluten-ut/spark33/src/test/scala/io/glutenproject/utils/velox/VeloxTestSettings.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -682,7 +682,6 @@ class VeloxTestSettings extends BackendTestSettings {
// decimal failed ut
.exclude("SPARK-34212 Parquet should read decimals correctly")
// Timestamp is read as INT96.
.exclude("SPARK-10634 timestamp written and read as INT64 - truncation")
.exclude("Migration from INT96 to TIMESTAMP_MICROS timestamp type")
.exclude("SPARK-10365 timestamp written and read as INT64 - TIMESTAMP_MICROS")
.exclude("SPARK-36182: read TimestampNTZ as TimestampLTZ")
Expand All @@ -698,7 +697,6 @@ class VeloxTestSettings extends BackendTestSettings {
// decimal failed ut
.exclude("SPARK-34212 Parquet should read decimals correctly")
// Timestamp is read as INT96.
.exclude("SPARK-10634 timestamp written and read as INT64 - truncation")
.exclude("Migration from INT96 to TIMESTAMP_MICROS timestamp type")
.exclude("SPARK-10365 timestamp written and read as INT64 - TIMESTAMP_MICROS")
.exclude("SPARK-36182: read TimestampNTZ as TimestampLTZ")
Expand Down
2 changes: 0 additions & 2 deletions gluten-ut/spark34/src/test/scala/io/glutenproject/utils/velox/VeloxTestSettings.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -668,7 +668,6 @@ class VeloxTestSettings extends BackendTestSettings {
// decimal failed ut
.exclude("SPARK-34212 Parquet should read decimals correctly")
// Timestamp is read as INT96.
.exclude("SPARK-10634 timestamp written and read as INT64 - truncation")
.exclude("Migration from INT96 to TIMESTAMP_MICROS timestamp type")
.exclude("SPARK-10365 timestamp written and read as INT64 - TIMESTAMP_MICROS")
.exclude("SPARK-36182: read TimestampNTZ as TimestampLTZ")
Expand All @@ -684,7 +683,6 @@ class VeloxTestSettings extends BackendTestSettings {
// decimal failed ut
.exclude("SPARK-34212 Parquet should read decimals correctly")
// Timestamp is read as INT96.
.exclude("SPARK-10634 timestamp written and read as INT64 - truncation")
.exclude("Migration from INT96 to TIMESTAMP_MICROS timestamp type")
.exclude("SPARK-10365 timestamp written and read as INT64 - TIMESTAMP_MICROS")
.exclude("SPARK-36182: read TimestampNTZ as TimestampLTZ")
Expand Down
Loading