Docs: update schema change capability #28200
Conversation
octavia-squidington-iii
added
the
area/documentation
|
|
||
| :::note | ||
| Airbyte checks for any changes in your source schema before every sync or once every 24 hours, whichever is more frequent. |
There was a problem hiding this comment.
@mfsiega-airbyte I think I recall this being the case, but am not sure now if this is true. Is it still once every 24 hours or does it check on sync start as well?
There was a problem hiding this comment.
Updated to say every 24 hours
There was a problem hiding this comment.
I would phrase it as: Airbyte checks for any changes in your source schema before syncing, at most once every 24 hours.
There was a problem hiding this comment.
Great, updated 👍
|
|
||
| When a new column is detected and propagated, values for that column will be filled in for the updated rows. If you are missing values for rows not updated, a backfill can be done by completing a full refresh. | ||
|
|
||
| When a column is deleted, the values for that column will stop updating for the updated rows and be filled with Null values. |
There was a problem hiding this comment.
I think that we will actually now delete these columns immediately, based on the related convo in Normalization.
There was a problem hiding this comment.
@alex-gron Was thinking about this more today after our chat - I agree that's the behavior we decided on. Does that mean the proposed changes will come with Destinations V2 - which I believe that's coming at the end of Q3? On the one hand, we could publish these with the foresight those changes will be coming, or re-publish with the updates when V2 is officially released. I'm leaning towards the latter but let me know if you think it's better to just update now!
There was a problem hiding this comment.
Sorry for the late reply! I'm good with that option :)
| @@ -69,3 +83,6 @@ In addition to Airbyte Cloud’s automatic schema change detection, you can manu | |||
| 3. In the **Activate the streams you want to sync** table, click **Refresh source schema** to fetch the schema of your data source. | |||
|
|
|||
| 2. If there are changes to the schema, you can review them in the **Refreshed source schema** dialog. | |||
There was a problem hiding this comment.
Should this be a 4. instead of 2. ?
There was a problem hiding this comment.
Oh nice! Our markdown system fixes this automatically as "4" but I've also updated it so it's clearer.
| Breaking schema changes occur when: | ||
| * The data type of a field from the source changes | ||
| * An existing primary key is removed from the source | ||
| * An existing cursor is removed from the source | ||
|
|
||
| To review and fix breaking schema changes: |
There was a problem hiding this comment.
What is the fix is the data type changes?
There was a problem hiding this comment.
From our chat I believe we decided to make any data type changes breaking changes. Not sure if that answers your question though - lmk if I missed something!
There was a problem hiding this comment.
Ah, I had a typo in my original question.
I meant to ask - what should a user do when they encounter a data type change? Is the only option in that case to run a reset?
* Schema update doc * Add note on notifications * update to 24 hr * update refresh language * removing data type changes
What
Updates docs to reflect latest capabilities of the schema change and management feature.