Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: restore queue model and consumer on disconnection #428

Merged
merged 1 commit into from
Nov 28, 2022

Conversation

sarmis
Copy link
Contributor

@sarmis sarmis commented Nov 24, 2022

We have seen cases where a failure on the load balancers in front of our rabbit clusters triggers disconnections from the rabbit cluster and thus deletions of the ephemeral queues. This MR tries to solve this issue by recreating the model, the queue, and the binding and adding a consumer. The old model is then disposed.

private void OnReconnect()
{
LogMessage("Bus Reconnected, flushing in-memory keys");
_localCache.Flush();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why flushing in-memory keys here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a disconnection happens (ie the load balancer between app and rabbit fails) then the queue will be deleted and any invalidations will be missed. I believe the correct approach is to flush the in-memory buffer in order to avoid having stale data.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not a good idea to flush the in memory caching here!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure that understand. Do you suggest to keep the (possibly stale) in-memory cache as is after a disconnection or do you have in mind a better way to refresh the in-memory cache?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you suggest to keep the (possibly stale) in-memory cache as is after a disconnection

Yes.

When a reconnection occurs, not all memory caches are expired, and if flush is executed at this time, it may cause application jitter.

In addition, all in-memory caches have an expiration time, and they are automatically removed after they expire.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, that's an understandable argument. Our use case requires however to be sure that the case is not stale. Would it be ok if we do either of these things:

a) Add an option to FlushOnReconnect (default: false)
b) Add a callback in order for the application to decide how to handle a reconnection

I would prefer (a) but I can do (b)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(a) is a good idea.
(b) is ok as well.

😄

Copy link
Member

@catcherwong catcherwong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@catcherwong catcherwong merged commit be302f9 into dotnetcore:dev Nov 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants