[#21252] Enforce pubsub message publishing limits in the python SDK

[scwhittle]

This improves visiblity as otherwise messages can become stuck publishing
inside runners such as dataflow that specialize pubsub publishing. This also allows users to handle errors by
catching them.

Fixes issue #21252

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[asf-ci]

Can one of the admins verify this patch?

[asf-ci]

Can one of the admins verify this patch?

[asf-ci]

Can one of the admins verify this patch?

[codecov]

Codecov Report

Merging #17794 (5ae4d01) into master (ca33943) will decrease coverage by 0.03%.
The diff coverage is 75.00%.

@@            Coverage Diff             @@
##           master   #17794      +/-   ##
==========================================
- Coverage   74.00%   73.96%   -0.04%     
==========================================
  Files         695      696       +1     
  Lines       91898    92205     +307     
==========================================
+ Hits        68013    68204     +191     
- Misses      22639    22755     +116     
  Partials     1246     1246              

Flag	Coverage Δ
python	83.64% <75.00%> (-0.11%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdks/python/apache_beam/io/gcp/pubsub.py	90.78% <75.00%> (+0.13%)	⬆️
...ache_beam/runners/dataflow/dataflow_job_service.py	50.00% <0.00%> (-12.17%)	⬇️
...eam/transforms/py_dataflow_distribution_counter.py	91.42% <0.00%> (-4.87%)	⬇️
sdks/python/apache_beam/dataframe/io.py	88.78% <0.00%> (-3.26%)	⬇️
...eam/runners/portability/fn_api_runner/fn_runner.py	87.51% <0.00%> (-2.50%)	⬇️
...nners/portability/fn_api_runner/worker_handlers.py	77.89% <0.00%> (-1.45%)	⬇️
sdks/python/apache_beam/io/source_test_utils.py	88.01% <0.00%> (-1.39%)	⬇️
sdks/python/apache_beam/utils/counters.py	85.39% <0.00%> (-1.36%)	⬇️
sdks/python/apache_beam/internal/dill_pickler.py	85.61% <0.00%> (-0.72%)	⬇️
...thon/apache_beam/ml/inference/sklearn_inference.py	92.50% <0.00%> (-0.19%)	⬇️
... and 24 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ca33943...5ae4d01. Read the comment docs.

[scwhittle]

R: @pabloem

[scwhittle]

@pabloem Friendly ping

[pabloem]

LGTM
@scwhittle could you please fix the test here? (https://ci-beam.apache.org/job/beam_PreCommit_Python_Commit/23449/testReport/junit/apache_beam.io.gcp.pubsub_test/TestPubsubMessage/test_payload_publish_invalid/)

and there's a rcommendation from lint: https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Commit/14277/console

[scwhittle]

Thanks Pablo, PTAL. There are a couple failures that appear unrelated though Run Python PreCommit hasn't finished yet

[pabloem]

Run Python 3.8 PostCommit

[pabloem]

Run Python 3.8 PostCommit

[scwhittle]

Run Python 3.8 PostCommit

[scwhittle]

Thanks for the test fix, sorry I wasn't able to get my virtualenv setup so I was having trouble running tests locally before pushing.

[pabloem]

Run Python 3.8 PostCommit

[pabloem]

Run Python 3.8 PostCommit

[pabloem]

postcomm running: https://ci-beam.apache.org/job/beam_PostCommit_Python38_PR/526/

[pabloem]

Run Portable_Python PreCommit

[pabloem]

Run Portable_Python PreCommit

[pabloem]

thanks Sam! merging. LGTM