[Epic] Next Steps for DBT / normalization

[cgardens]

Tell us about the problem you're trying to solve

Here is a compiled list of features for normalization from discussions/feedbacks with users

• Handling Source Schema Changes
• I want to run another process at the end of the sync (Post-hooks)
• Closing the DBT loop back to Airbyte
• Customize Airbyte transformation step (with/without DBT)
• Incremental batch normalization
• Configure Normalization options per stream on connection page UI
• DBT Lint/Tests/Expectations before handing data off
• Plug-N-Play with popular DBT packages
• Various opened Issues:
  • Support syncing to time-partitioned tables in BigQuery: Support syncing to time-partitioned tables in BigQuery #2579 https://discourse.getdbt.com/t/benchmarking-incremental-strategies-on-bigquery/981
  • Nested columns loaded as Struct in BigQuery: Load JSON data into BigQuery as structured data (records with repeated and nested fields, using STRUCT types) #1927
  • Nested columns in the parent table

Describe the solution you’d like

Google document to discuss and comment on these ideas in more details which would serve to prioritize roadmap around transformations and re-order the items

┆Issue is synchronized with this Asana task by Unito

[ChristopheDuong]

All discussions are written in this doc with more details, please feel free to comment!!

Anyone should be able to access:
https://docs.google.com/document/d/1_eii32qoznzQ7_7aSzI5S0b-tajCeHbgRyQvYwNjK40/edit?usp=sharing

[leecheeaun]

Upvote for Incremental batch normalization - this was what I was looking for in 2683.

The rationale for this as a user is to offload querying the OLTP DB to querying data warehouses, or to avoid using OLTP DBs as warehouse.

Also, with regards to incremental batch normalization, looking for low latency querying difference between source and destination, so this would mean the time taken for the data to be normalized in the warehouse.

My use case can best be illustrated below, where I have a (relatively) large amount of historical data, and a few rows of new data added per day to an OLTP DB

[hoanghapham]

Upvote for better Handling Source Schema Changes. Currently, when I want to add a new table to be synced, Airbyte cleans all of the existing destination tables, which is not a desired behavior.

[ChristopheDuong]

Upvote for better Handling Source Schema Changes. Currently, when I want to add a new table to be synced, Airbyte cleans all of the existing destination tables, which is not a desired behavior.

It seems like the related issue to track this is here: #3520

[fixico-abdel]

Upvote for better Handling Source Schema Changes. Currently, when I want to add a new table to be synced, Airbyte cleans all of the existing destination tables, which is not a desired behavior.

[erichartono]

Upvote for this improvement. I had a sync with incremental dedupe, and it took 48 minutes on normalisation process.

[ChristopheDuong]

The dedicated ticket for incremental normalization support is #4286