Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Missed Updates Over Multiple Syncs #98

Merged
merged 3 commits into from
Aug 23, 2019

Conversation

dmosorast
Copy link
Contributor

@dmosorast dmosorast commented Aug 22, 2019

In this commit, I added a lookback window to engagements. There's another case where this same sort of thing can happen due to the way the companies API endpoint works (same as engagements, e.g., no sorting, no query-by-updated-at, etc.).

However, depending on the length of each tap-run, the existing pattern won't work across multiple runs, since it doesn't save any state until the end of the stream's sync.

The Solution Presented Here
The original pattern was to calculate a lookback window consisting of the end - start of the most recent sync. However, this PR modifies that to forcing the bookmark to be the min(current_sync_start, max_bk_seen) at the end of a sync, and saving current_sync_start into the state.

This is equivalent, and because current_sync_start is present in the state, it works across multiple interrupted invocations of the tap, so it should be more resilient to failure.

@dmosorast dmosorast merged commit dbcd32c into master Aug 23, 2019
@dmosorast dmosorast deleted the fix/multi-sync-missed-updates branch August 23, 2019 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant