Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Question] Is there any way to sync with results of async operations? #2455

Closed
abyss7 opened this issue Aug 5, 2019 · 3 comments
Closed

Comments

@abyss7
Copy link

abyss7 commented Aug 5, 2019

The operations of interest are:

  • pause/resume
  • subscribe/unsubscribe

The related calls to rd_kafka_resume_partitions(), rd_kafka_pause_partitions(), rd_kafka_subscribe(), etc. just put a task into some queue which is processed by another thread async. I can't find any high-level mechanism (like future-promise) to sync with a result of these operations, and can't figure out how to check if operation is completed within a couple lines of code.

The basic scenario:

  1. Check if consumer is subscribed to anything.
  2. If not, then pause all consumption - to not accidentally read any messages while polling for subscription messages. Wait for pause to apply.
  3. Subscribe and poll with timeout until anything gets assigned.
  4. Resume and wait for it - especially if subscription gone wrong and failed.
@abyss7
Copy link
Author

abyss7 commented Aug 5, 2019

The better description of high-level scenario.

If consumer is not subscribed (or assigned) to anything, then subscribe it in sync way, make sure it's resumed, and make sure that next call to poll() will return new messages if there are any on broker-side.

@edenhill
Copy link
Contributor

pause/resume..() are currently asynchronous, as you say, but it is probably a good idea to make them synchronous.

edenhill added a commit that referenced this issue Aug 14, 2019
This makes sure that a consumer_poll() call after pause() will not
return any messages.
edenhill added a commit that referenced this issue Aug 16, 2019
This makes sure that a consumer_poll() call after pause() will not
return any messages.
edenhill added a commit that referenced this issue Aug 16, 2019
This makes sure that a consumer_poll() call after pause() will not
return any messages.
@edenhill
Copy link
Contributor

edenhill commented Apr 7, 2022

Fixed

@edenhill edenhill closed this as completed Apr 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants