Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add discovered catalog refresh functionality #1000

Closed
sherifnada opened this issue Nov 17, 2020 · 4 comments
Closed

Add discovered catalog refresh functionality #1000

sherifnada opened this issue Nov 17, 2020 · 4 comments
Labels

Comments

@sherifnada
Copy link
Contributor

sherifnada commented Nov 17, 2020

See #996 for context

In the scheduler, add a process which runs discovery then performs one of the following actions:

  • If the catalog was amended: move the connection to a new state DEFUNCT_CATALOG_AMENDED in the backend. Connections whose source is in this state do not sync data.
  • If the catalog has expanded/contracted and not amended, we update the source's catalog selection by adding any new streams/fields without selecting them. If any streams/fields were removed by the contraction, remove them from the catalog (even if they were selected).

┆Issue is synchronized with this Asana task by Unito

@sherifnada sherifnada added the type/enhancement New feature or request label Nov 17, 2020
@ChristopheDuong
Copy link
Contributor

ChristopheDuong commented Nov 17, 2020

Small question: to detect that catalogs changed (amended/expanded or contracted), are they going to be persisted/stored somewhere? if yes, where?

Is it going to be handled in a similar way to states?
#990

@ChristopheDuong
Copy link
Contributor

As a side note for the future, it would be really great, if we can send notifications (slack, email, etc) when such changes happen to the catalogs...

So that user can decide to go select or un-select new fields/streams or be aware that the Analytics pipelines downstream may get affected (if streams/fields were removed, then some expected data could stop being replicated/refreshed).

@sherifnada
Copy link
Contributor Author

@ChristopheDuong love the notifications idea! the infrastructure for that would also get us closer to having an "audit log" of schema changes for debuggability

FWIW this issue is getting bumped out of the next release, see #996 for more context

@evantahler
Copy link
Contributor

Closing in favor of the epic #14278

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants