Reverted email analytics jobs commits #20835
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
9larsons
added a commit
that referenced
this pull request
Aug 27, 2024
ref https://linear.app/tryghost/issue/ENG-1518 After releasing the analytics job improvements, it appears for large sites we're awfully close to missing some Mailgun events because of an unexpected behavior of the aggregateStats call for just the opened events job. This is taking 2-5x(+) the amount of time that the aggregate queries take for the other jobs, despite not being dependent on the events. To err on the side of caution, we're going to roll this back and look to optimize the aggregation queries before re-implementing. And we may be a bit more cautious in giving _some_ but not _all_ priority to the `opened` events.
9larsons
added a commit
that referenced
this pull request
Sep 5, 2024
ref #20835 - reimplemented email analytics changes that prioritized opened events over other events in order to speed up open analytics - added db persistence to fetch missing job to ensure we re-fetch every window of events, especially important if we restart following a large email batch We learned a few things with the previous trial run of this. Namely, that event throughput is not as high as we initially saw in the data for particularly large databases. This set of changes is more conservative, while a touch more complicated, in ensuring we capture edge cases for really large newsletter sends (100k+ members). In general, we want to make sure we're fetching new open events at least every 5 mins, and often much faster than that, unless it's a quiet period (suggesting we haven't had a newsletter send or much outstanding event data).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ref https://linear.app/tryghost/issue/ENG-1518
After releasing the analytics job improvements, it appears for large sites we're awfully close to missing some Mailgun events because of an unexpected behavior of the aggregateStats call for just the opened events job. This is taking 2-5x(+) the amount of time that the aggregate queries take for the other jobs, despite not being dependent on the events.
To err on the side of caution, we're going to roll this back and look to optimize the aggregation queries before re-implementing. And we may be a bit more cautious in giving some but not all priority to the
openedevents.