Auto-detect and migrate new schemas in source DB

[sjlogan]

I am trying to move data from a multi tenant application to a data warehouse. In the source database, each tenant has its own db schema and tables. All of the tables in each tenant's schema match the other schemas. As we add new tenants, we need to manually configure new migrations (or update existing ones) in whatever EL(T) tool we use.

It would be AMAZING if an EL(T) tool could auto-detect new schemas in the source db and automatically migrate all of the data in them to the destination. It would be MORE AMAZING if an EL(T) tool could auto-detect new schemas in the source db and create a matching migration configuration to an existing schema and then use to automatically start data migration.

I could also see a world where an Airbyte API could be triggered on our end when we create a new schema in our source DB that allows me to create a matching migration configuration to an existing schema.

The only way I have been able to get around this problem is manually configuring existing migration logic in other tools to account for each new schema in the source db.

┆Issue is synchronized with this Asana task by Unito

[cgardens]

Hi @sjlogan everything you've asked for here makes sense to me! This is definitely in line with our vision. We will keep you posted as we iterate towards these goals. Our focus is on pushing out incremental syncs as a feature now, but schema migration is a top priority after that.

[marcosmarxm]

This was solved by #17520