Rewriting IncrementalPublishingKafkaIndexTaskRunner

[jihoonson]

Kafka indexing service has been being actively developed since it was first introduced. As requirements increase, a ton of new cool features have also been added for Kafka indexing service like #4815. However, the active development led frequent code changes and introduced more bugs than before. (See this discussion for 0.12.2 release. The release has postponed to fix more bugs in kafka indexing service.)

I think one of the reasons causing more bugs is its code complexity. It is very complicated so that people should read the entire class (1700 lines) again and again to exactly understand how it works, at least for me. This complex code make it difficult to make kafka index task more robust by handling all possible exceptional cases. So, I think it wouldn't be easy to make kafka indexing service more stable with the current implementation.

KafkaIndexTask changes its state based on the events from the kafka supervisor, so basically I think we can do better by adopting the event-driven architecture to it.
I have been working on fixing some bugs as well as some simple refactoring issues for kafka indexing service (like #5854), and am also available for this issue. Actually, I really want to do this work.

[gianm]

Definitely in favor of a rethink. You can see layers of older designs cluttering the landscape of the kafka task and supervisor, and it'd be good to clear those out.

What were you thinking in terms of a sketch of a design for a newer, clearer architecture?

[jihoonson]

I'm removing this from 0.13.0 milestone.

[github-actions]

This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the [email protected] list.
Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.