Partitioned sources revoke/assign ignore re-assigned partitions

[ennru]

#382 reported lost messages on partition reassignment. This was solved (or at least improved) with #462 and #517, by delaying the actual handling of the revoked topic partitions: SubSourceLogic.partitionRevokedCB

This does not work correctly whenever the Kafka client issues revoke and assign signals for the same partitions within the timeframe of waitClosePartition. Whenever that happens any assigned partitions - that were marked as revoked just before - will be ignored: SubSourceLogic.partitionAssignedCB.

[ennru]

This issue becomes visible when using manual offsets via plainPartitionedManualOffsetSource.

[tkroman]

See also #539. AFAIU #590 only works for external storage, right? I’m still willing to put some work into my original issue if I had some pointers on how to approach this, please let me know if there is anything I can do.

[ennru]

@tkroman Thanks for pointing this out. Yes, #590 just changed what partitions are passed to the function fetching offsets from external storage.

[Camsteack]

@ennru Thanks a lot for the quick fix

[2m]

If you would like to try out a snapshot version with this fix merged in, we have just released a 0.22+75-ada5b3a2 to the snapshots repo.

[Camsteack]

Will do !