use sync start time to bookmark contacts stream #226
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of change
For following streams, stream records were missed which were updated during stream was syncing.
Implemented a fix to bookmark minimum of stream sync start time and max. modifiedTime.
e.g.
Consider a scenario,
max. modified_time=99
)max. modified_time=103
)vid_50
andvid_250
got updated with 106 and 108 respectively.max. modified_time=108
)With existing implementation, after completion of sync, we set
bookmark_value=108
. This will miss the recordvid_50
updated at 106 in next sync.But if we set
bookmark_value=min(sync_start_time=100, max_modified_time=108)
i.e. 100, in next syncvid_50
record will be synced, along with duplicatevid_250
record.Manual QA steps
_sync_contact_vids()
for first 100 vids. Then updated one record from first batch of 100 vids (vid=50
)and one record from next batch of vids (vid=150
).sync_contacts()
completed bookmark set is equal tosync_start_time
which less than max modified time.contacts
records were synced i.e.vids=[50, 150]
.modified_time
betweenvids=[50, 150]
Risks
contacts
stream records.Rollback steps