[HUDI-7303] Fix date field type unexpectedly convert to Long when usi… #10517
Conversation
| case TIME_WITHOUT_TIME_ZONE: | ||
| return predicateSupportsLtGt(functionDefinition, intColumn(columnName), (Integer) literal); | ||
| case BIGINT: | ||
| case DATE: | ||
| case TIMESTAMP_WITHOUT_TIME_ZONE: |
There was a problem hiding this comment.
Is there any exception thrown here?
There was a problem hiding this comment.
Hi @danny0405 ,
As far as I could see there should be no exceptions thrown here. I have gone through all TPCDS queries. They could work well with all patches applied that I sumbitted recently.
If there might be some potential risks for this change, please let me know. Thanks.
There was a problem hiding this comment.
Then why we need this fix?
There was a problem hiding this comment.
Hi @danny0405 ,
Please see: HUDI-7303.
In parquet, date type is stored as INT32 (epoch day). But if we add some conditions with date typed field in SQL where clause, its type will unexpectedly convert to Long.
|
There are test failures: [ERROR] Errors:
[ERROR] TestHoodieTableSource.testBucketPruningSpecialKeyDataType:267 » ClassCast java...
[ERROR] TestHoodieTableSource.testBucketPruningSpecialKeyDataType:267 » ClassCast java...
|
Err... I could not find exceptions in CI check reports. Could you please help me about the detailed error stacktrace? Thanks. |
…ng date comparison operator.
…ng date comparison operator (#10517)
…ng date comparison operator.
Change Logs
When using between, less than (less than or equal) or greater than (greater than or equal) operators with field typed of date, the date type will unexpected convert to Long, which is incompatible with its primitive type INT32.
Impact
No impact.
Risk level (write none, low medium or high below)
Low risk level.
Documentation Update
No need to update the documentation.
Contributor's checklist