[Improve][HiveSink]Fix Partition loading failed. #3886
Conversation
|
@EricJoy2048 @Hisoka-X @TaoZex please review .thanks. |
There was a problem hiding this comment.
The best solution is to limit the user's ability to set partition_name_expression not add a new config option.
cc @EricJoy2048
|
@EricJoy2048 @Hisoka-X @TaoZex Please see if this pr is reasonable. In fact, there are such use scenarios. |
|
If does not have a partition name can't be loaded, why we should allow it? Maybe check |
ok, If it is unreasonable, I will close it. |
No need, can your add the code in this PR to check |
@Hisoka-X @TyrantLucifer PTAL, partition_dir_expression parameter adds the validation of the partition name to prevent the partition from failing to load |
…the partition loading method is different to ensure successful partition loading
…on name to prevent the partition from failing to load
…on name to prevent the partition from failing to load
…it of partition expression
TyrantLucifer
force-pushed
the
hive-partition-name-no-contaion
branch
from
e178bf8 to
f00aa1f
Compare
|
cc @Hisoka-X |
* Whether the partition directory contains the partition name. If not, the partition loading method is different to ensure successful partition loading * partition_dir_expression parameter adds the validation of the partition name to prevent the partition from failing to load * update change log. * Fix the problem of duplicate hive partition loading. * partition_dir_expression parameter adds the validation of the partition name to prevent the partition from failing to load * revert * revert * [Improve][Connector-V2][Hive] Improve config check logic, add the limit of partition expression Co-authored-by: zhaoliang01 <[email protected]> Co-authored-by: tyrantlucifer <[email protected]>
1.If user sets 'partition_dir_expression = "${v0}"' in hive-sink,it means that the partition directory does not have a partition name. Then the partition loading will fail.
2.Like this : /home/test/warehouse/test.db/tmp_test_day/20230106
3.A new loading method is added for this case to ensure successful loading of partitions.
Purpose of this pull request
Check list
New License Guide
release-note.