Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve stream sourcing consumers stall handling #5366

Merged
merged 11 commits into from
Apr 29, 2024

Conversation

jnmoyne
Copy link
Contributor

@jnmoyne jnmoyne commented Apr 29, 2024

  • Resets the stalling detection timestamp upon re-creating the consumer
  • Adjust the idle heart beat for source/mirror consumers from 1s to 5s

jnmoyne added 11 commits April 27, 2024 13:49
- Reset the stall detection timestamp when re-creating a new consumer to avoid the consumer then being detected stale after just 1 HB period (rather than 3).
- retryMirrorConsumer should call cancelMirrorConsumer to clean up any existing consumer before re-creating a new consumer.
- Adjust `sourceHealthCheckInterval` from 1 to 20 seconds, which does not trigger any stalling detections anymore for the SourcingMany test.

Signed-off-by: Jean-Noël Moyne <[email protected]>
Signed-off-by: Jean-Noël Moyne <[email protected]>
Adjust the check at the end of `TestJetStreamClusterMirrorAndSourceExpiration` to 40s (as it has to be slightly more than 3 HB interval)

Signed-off-by: Jean-Noël Moyne <[email protected]>
Signed-off-by: Jean-Noël Moyne <[email protected]>
@jnmoyne jnmoyne requested a review from a team as a code owner April 29, 2024 19:44
Copy link
Member

@derekcollison derekcollison left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@derekcollison derekcollison merged commit 3be817c into main Apr 29, 2024
4 checks passed
@derekcollison derekcollison deleted the jnm/sourcing-scaling-test branch April 29, 2024 20:14
wallyqs pushed a commit that referenced this pull request May 1, 2024
- Resets the stalling detection timestamp upon re-creating the consumer
- Adjust the idle heart beat for source/mirror consumers from 1s to 5s

---------

Signed-off-by: Jean-Noël Moyne <[email protected]>
wallyqs pushed a commit that referenced this pull request May 1, 2024
- Resets the stalling detection timestamp upon re-creating the consumer
- Adjust the idle heart beat for source/mirror consumers from 1s to 5s

---------

Signed-off-by: Jean-Noël Moyne <[email protected]>
wallyqs added a commit that referenced this pull request May 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants