Skip to content

Commit

Permalink
[SPARK-50870][SQL] Add the timezone when casting to timestamp in V2Sc…
Browse files Browse the repository at this point in the history
…anRelationPushDown

### What changes were proposed in this pull request?

Add the timezone information to a cast expression when the destination type requires it.

### Why are the changes needed?

When current_timestamp() is materialized as a string, the timezone information is gone (e.g., 2024-12-27 10:26:27.684158) which prohibits further optimization rules from being applied to the affected data source.

For example,

```
Project [1735900357973433#10 AS current_timestamp()#6]
+- 'Project [cast(2025-01-03 10:32:37.973433#11 as timestamp) AS 1735900357973433#10]
   +- RelationV2[2025-01-03 10:32:37.973433#11] xxx
```

-> This query fails to execute because the injected cast expression lacks the timezone information.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Existing tests.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #49549 from changgyoopark-db/SPARK-50870.

Authored-by: changgyoopark-db <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit 24abb0f)
Signed-off-by: Wenchen Fan <[email protected]>
  • Loading branch information
changgyoopark-db authored and cloud-fan committed Jan 21, 2025
1 parent 5397e12 commit 12d0799
Showing 1 changed file with 6 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -328,7 +328,12 @@ object V2ScanRelationPushDown extends Rule[LogicalPlan] with PredicateHelper {
if (expression.dataType == expectedDataType) {
expression
} else {
Cast(expression, expectedDataType)
val cast = Cast(expression, expectedDataType)
if (cast.timeZoneId.isEmpty && cast.needsTimeZone) {
cast.withTimeZone(conf.sessionLocalTimeZone)
} else {
cast
}
}

def buildScanWithPushedAggregate(plan: LogicalPlan): LogicalPlan = plan.transform {
Expand Down

0 comments on commit 12d0799

Please sign in to comment.