You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, we have a requirement for source record positions, which is that they have to be unique across all sources in a pipeline.
Internally, that's used to make acknowledgments work (an acker node caches messages, with record positions being the keys; when a destination acks a record, the respective message is found in the cache using the position and ack'd).
This limitation mean that source connector developers need to make sure that the positions can not only be used to resume reading from a source, but also that those positions are globally unique, so that the source connectors can be used in any combination. While that is possible and we can make it clear, it would also be good to get away with this limitation and not make developers worry about it.
The text was updated successfully, but these errors were encountered:
This is going to impact a pipeline that has multiple sources. While it may be unlikely for two sources to have the same positions, it would result in unexpected behavior and would have errors. A current workaround would be to have separate pipelines for each upstream source connector. Not ideal but a workaround until this can be fixed.
Feature description
Currently, we have a requirement for source record positions, which is that they have to be unique across all sources in a pipeline.
Internally, that's used to make acknowledgments work (an acker node caches messages, with record positions being the keys; when a destination acks a record, the respective message is found in the cache using the position and ack'd).
This limitation mean that source connector developers need to make sure that the positions can not only be used to resume reading from a source, but also that those positions are globally unique, so that the source connectors can be used in any combination. While that is possible and we can make it clear, it would also be good to get away with this limitation and not make developers worry about it.
The text was updated successfully, but these errors were encountered: