Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove limitation that source position need to be globally unique #517

Closed
hariso opened this issue Jul 8, 2022 · 2 comments
Closed

Remove limitation that source position need to be globally unique #517

hariso opened this issue Jul 8, 2022 · 2 comments
Assignees
Labels
feature New feature or request

Comments

@hariso
Copy link
Contributor

hariso commented Jul 8, 2022

Feature description

Currently, we have a requirement for source record positions, which is that they have to be unique across all sources in a pipeline.

Internally, that's used to make acknowledgments work (an acker node caches messages, with record positions being the keys; when a destination acks a record, the respective message is found in the cache using the position and ack'd).

This limitation mean that source connector developers need to make sure that the positions can not only be used to resume reading from a source, but also that those positions are globally unique, so that the source connectors can be used in any combination. While that is possible and we can make it clear, it would also be good to get away with this limitation and not make developers worry about it.

@hariso hariso added feature New feature or request triage Needs to be triaged labels Jul 8, 2022
@meroxa-machine meroxa-machine moved this to Triage in Conduit Main Jul 8, 2022
@neovintage
Copy link
Contributor

This is going to impact a pipeline that has multiple sources. While it may be unlikely for two sources to have the same positions, it would result in unexpected behavior and would have errors. A current workaround would be to have separate pipelines for each upstream source connector. Not ideal but a workaround until this can be fixed.

@neovintage neovintage removed the triage Needs to be triaged label Jul 18, 2022
@neovintage neovintage removed the status in Conduit Main Jul 18, 2022
This was referenced Jul 22, 2022
@lovromazgon lovromazgon self-assigned this Jul 26, 2022
@lovromazgon lovromazgon moved this to In Review in Conduit Main Jul 26, 2022
@lovromazgon
Copy link
Member

Done in #522

Repository owner moved this from In Review to Done in Conduit Main Jul 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
Archived in project
Development

No branches or pull requests

3 participants