187: Adding apache hudi support to dbt#210

Merged

jtcohen6 merged 24 commits intodbt-labs:mainfrom vingov:apache_hudi_support

Nov 19, 2021

+261-8

Commits on Oct 26, 2021

initial working version

Vinoth Govindarajan
authored and
vingov
committed
Rebased and resolve all the merge conflicts.

Vinoth Govindarajan
authored and
vingov
committed
Rebased and resolved merge conflicts.

Vinoth Govindarajan
authored and
vingov
committed
Removed hudi dep jar and used the released version via packages option

Vinoth Govindarajan
authored and
vingov
committed
Added insert overwrite unit tests for hudi

Vinoth Govindarajan
authored and
vingov
committed
Used unique_key as default value for hudi primaryKey option

Vinoth Govindarajan
authored and
vingov
committed
Updated changelog.md with this new update.

Vinoth Govindarajan
authored and
vingov
committed
Final round of testing and few minor fixes

Vinoth Govindarajan
authored and
vingov
committed
Fixed lint issues

Vinoth Govindarajan
authored and
vingov
committed
Fixed the integration tests

Vinoth Govindarajan
authored and
vingov
committed
Fixed the circle ci env to add hudi packages

Vinoth Govindarajan
authored and
vingov
committed
Updated hudi spark bundle to use scala 2.11

Vinoth Govindarajan
authored and
vingov
committed
Fixed Hudi incremental strategy integration tests and other integration tests

Vinoth Govindarajan
authored and
vingov
committed
Fixed the hudi hive sync hms integration test issues

Vinoth Govindarajan
authored and
vingov
committed
Added sql HMS config to fix the integration tests.

Vinoth Govindarajan
authored and
vingov
committed
Added hudi hive sync mode conf to CI

Vinoth Govindarajan
authored and
vingov
committed
Set the hms schema verification to false

Vinoth Govindarajan
authored and
vingov
committed
Removed the merge update columns hence its not supported.

Vinoth Govindarajan
authored and
vingov
committed
Passed the correct hiveconf to the circle ci build script
vingov
committed
Disabled few incremental tests for spark2 and reverted to spark2 config
vingov
committed
Added hudi configs to the circle ci build script
vingov
committed

Commits on Nov 17, 2021

Commented out the Hudi integration test until we have the hudi 0.10.0 version
Vinoth Govindarajan
committed

Commits on Nov 18, 2021

Fixed the macro which checks the table type.
Vinoth Govindarajan
committed
Disabled this model since hudi is not supported in databricks runtime, will be added later
Vinoth Govindarajan
committed