Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use sync start time to bookmark contacts stream #226

Merged
merged 3 commits into from
Apr 26, 2023

Conversation

RushiT0122
Copy link
Contributor

@RushiT0122 RushiT0122 commented Apr 19, 2023

Description of change

For following streams, stream records were missed which were updated during stream was syncing.

  • contacts
  • contact_lists
  • deals
  • forms
  • owners
  • tickets
  • workflows

Implemented a fix to bookmark minimum of stream sync start time and max. modifiedTime.

e.g.

Consider a scenario,

  1. 1st sync started at 100 for vids 1-100 and completed in 5 seconds. (max. modified_time=99)
  2. 2nd sync started at 105 for vids 101-200 and completed in 5 seconds. (max. modified_time=103)
  3. While 2nd sync was in-progress, vid_50 and vid_250 got updated with 106 and 108 respectively.
  4. Last sync started at 110 for vids 201-300 and completed in 5 seconds. (max. modified_time=108)

With existing implementation, after completion of sync, we set bookmark_value=108. This will miss the record vid_50 updated at 106 in next sync.

But if we set bookmark_value=min(sync_start_time=100, max_modified_time=108) i.e. 100, in next sync vid_50 record will be synced, along with duplicate vid_250 record.

Manual QA steps

  • Simulated scenario discussed above manually by pausing sync after _sync_contact_vids() for first 100 vids. Then updated one record from first batch of 100 vids (vid=50)and one record from next batch of vids (vid=150).
  • Verified after sync_contacts() completed bookmark set is equal to sync_start_time which less than max modified time.
  • Verified that in next sync, only those 2 contacts records were synced i.e. vids=[50, 150].
  • Verified bookmark set to max. modified_time between vids=[50, 150]

Risks

  • In normal scenario, we should see least duplication of contacts stream records.
  • But if records are getting updated very frequently, then we may see noticeable amount of duplicate records.

Rollback steps

  • revert this branch

@RushiT0122 RushiT0122 marked this pull request as ready for review April 19, 2023 17:16
* update bookmarking for records updated between sync
* fix bookmarking and interrupted sync integration tests
* update unit test

---------

Co-authored-by: RushiT0122 <[email protected]>
@RushiT0122 RushiT0122 requested review from bryantgray and removed request for luandy64 April 25, 2023 15:58
@kethan1122 kethan1122 merged commit 8fa6a42 into master Apr 26, 2023
@kethan1122 kethan1122 deleted the update-contacts-bookmarking-strategy branch April 26, 2023 05:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants