Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dataflow runner specialization of PubsubIO should validate messages #21252

Closed
damccorm opened this issue Jun 4, 2022 · 5 comments
Closed

Dataflow runner specialization of PubsubIO should validate messages #21252

damccorm opened this issue Jun 4, 2022 · 5 comments
Assignees

Comments

@damccorm
Copy link
Contributor

damccorm commented Jun 4, 2022

Currently invalid messages, such as those with attributes exceeding the maximum size are processed by the bundle successfully but fail to commit.

Throwing an exception when trying to write such a message directly would increase visibility as well as allowing users to catch and handle such exceptions.

Imported from Jira BEAM-12942. Original Jira may contain additional context.
Reported by: scwhittle.

@damccorm
Copy link
Contributor Author

damccorm commented Jun 4, 2022

Unable to assign user @scwhittle. If able, self-assign, otherwise tag @damccorm so that he can assign you. Because of GitHub's spam prevention system, your activity is required to enable assignment in this repo.

@scwhittle
Copy link
Contributor

@damccorm Hi Danny, can you assign this to me? I don't seem to be able to though I have committed to the repo before.

@damccorm
Copy link
Contributor Author

damccorm commented Jun 7, 2022

One of the unfortunate downsides of moving to issues is that non-committers can't self-assign/triage issues that they didn't create by default. I think that's a workflow we'll need to support going forward though, I'll look into it. For the moment, I'd recommend commenting/declaring intent to work on any issues. Once #21719 is in, you should be able to self assign and close issues with chat-op commands

scwhittle added a commit to scwhittle/beam that referenced this issue Jun 10, 2022
… SDK

This improves visiblity as otherwise messages can become stuck publishing
inside the dataflow runner. This also allows users to handle errors by
catching them.

Fixes issue apache#21252 for python sdk.
pabloem added a commit that referenced this issue Jun 13, 2022
…hing limits in the python SDK

* [#21252] Enforce pubsub message publishing limits in the python SDK

This improves visiblity as otherwise messages can become stuck publishing
inside the dataflow runner. This also allows users to handle errors by
catching them.

Fixes issue #21252 for python sdk.

* fixtest

Co-authored-by: Pablo <[email protected]>
@scwhittle scwhittle removed their assignment Jun 13, 2022
@scwhittle
Copy link
Contributor

.take-issue

@scwhittle
Copy link
Contributor

.close-issue

bullet03 pushed a commit to akvelon/beam that referenced this issue Jun 20, 2022
…ssage publishing limits in the python SDK

* [apache#21252] Enforce pubsub message publishing limits in the python SDK

This improves visiblity as otherwise messages can become stuck publishing
inside the dataflow runner. This also allows users to handle errors by
catching them.

Fixes issue apache#21252 for python sdk.

* fixtest

Co-authored-by: Pablo <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants