Fix Missed Updates Over Multiple Syncs #98
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In this commit, I added a lookback window to engagements. There's another case where this same sort of thing can happen due to the way the companies API endpoint works (same as engagements, e.g., no sorting, no query-by-updated-at, etc.).
However, depending on the length of each tap-run, the existing pattern won't work across multiple runs, since it doesn't save any state until the end of the stream's sync.
The Solution Presented Here
The original pattern was to calculate a lookback window consisting of the
end - start
of the most recent sync. However, this PR modifies that to forcing the bookmark to be themin(current_sync_start, max_bk_seen)
at the end of a sync, and savingcurrent_sync_start
into the state.This is equivalent, and because
current_sync_start
is present in the state, it works across multiple interrupted invocations of the tap, so it should be more resilient to failure.