[Do not merge] Beam 2.58.0rc1 validation

[Abacn]

No description provided.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 7 lines in your changes missing coverage. Please review.

Project coverage is 42.32%. Comparing base (d7db191) to head (6d883be).

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1738      +/-   ##
============================================
- Coverage     42.32%   42.32%   -0.01%     
  Complexity     3182     3182              
============================================
  Files           794      794              
  Lines         46244    46246       +2     
  Branches       4951     4953       +2     
============================================
  Hits          19572    19572              
- Misses        25074    25076       +2     
  Partials       1598     1598              

Components	Coverage Δ
spanner-templates	63.70% <ø> (ø)
spanner-import-export	64.44% <ø> (ø)
spanner-live-forward-migration	74.97% <ø> (ø)
spanner-live-reverse-replication	51.84% <ø> (ø)
spanner-bulk-migration	82.79% <ø> (ø)

Files	Coverage Δ
...cloud/teleport/v2/templates/SpannerToBigQuery.java	0.00% <0.00%> (ø)

[Abacn]

Other than Yaml/PythonUDF templates, the following tests failing

SpannerToBigQueryIT.testSpannerToBigQuery:

SpannerToBigQueryIT.testSpannerToBigQueryNoSchemaFile

Error message:

java.lang.IllegalArgumentException: Both query and table cannot be specified at the same time for SpannerIO.read().
2024-07-19 12:25:55.130 EDT
	at org.apache.beam.sdk.io.gcp.spanner.SpannerIO$Read.expand(SpannerIO.java:890)
2024-07-19 12:25:55.130 EDT
	at org.apache.beam.sdk.io.gcp.spanner.SpannerIO$Read.expand(SpannerIO.java:708)

due to apache/beam#31570 . This is a breaking change. Currently for SpannerToBigQueryOptions both getSpannerTableId and getSqlQuery are not optional. Arguably this would break all users using the same template parameters for new beam version.

[Abacn]

SpannerIO config fixed by updating templates. Besides, YamlTemplate failed launch, error message

AttributeError: 'str' object has no attribute 'items'

 Traceback (most recent call last):
 File "/template/main.py", line 69, in <module>
 run()
 File "/template/main.py", line 63, in run
 main.run(argv=_preparse_jinja_flags(argv))
 File "/usr/local/lib/python3.11/site-packages/apache_beam/yaml/main.py", line 143, in run
 yaml_transform.expand_pipeline(
 File "/usr/local/lib/python3.11/site-packages/apache_beam/yaml/yaml_transform.py", line 1062, in expand_pipeline
 validate_against_schema(pipeline_spec, validate_schema)
 File "/usr/local/lib/python3.11/site-packages/apache_beam/yaml/yaml_transform.py", line 87, in validate_against_schema
 jsonschema.validate(pipeline, pipeline_schema(strictness))
 File "/usr/local/lib/python3.11/site-packages/jsonschema/validators.py", line 1330, in validate
 error = exceptions.best_match(validator.iter_errors(instance))
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/jsonschema/exceptions.py", line 475, in best_match
 best = next(errors, None)
 ^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/jsonschema/validators.py", line 384, in iter_errors
 for error in errors:
 File "/usr/local/lib/python3.11/site-packages/jsonschema/_keywords.py", line 296, in properties
 yield from validator.descend(
 File "/usr/local/lib/python3.11/site-packages/jsonschema/validators.py", line 432, in descend
 for error in errors:
 File "/usr/local/lib/python3.11/site-packages/jsonschema/_legacy_keywords.py", line 135, in items_draft6_draft7_draft201909
 yield from validator.descend(item, items, path=index)
 File "/usr/local/lib/python3.11/site-packages/jsonschema/validators.py", line 432, in descend
 for error in errors:
 File "/usr/local/lib/python3.11/site-packages/jsonschema/_keywords.py", line 275, in ref
 yield from validator._validate_reference(ref=ref, instance=instance)
 File "/usr/local/lib/python3.11/site-packages/jsonschema/validators.py", line 432, in descend
 for error in errors:
 File "/usr/local/lib/python3.11/site-packages/jsonschema/_keywords.py", line 383, in if_
 if validator.evolve(schema=if_schema).is_valid(instance):
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/jsonschema/validators.py", line 500, in is_valid
 error = next(self.iter_errors(instance), None)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/jsonschema/validators.py", line 384, in iter_errors
 for error in errors:
 File "/usr/local/lib/python3.11/site-packages/jsonschema/_keywords.py", line 294, in properties
 for property, subschema in properties.items():
 ^^^^^^^^^^^^^^^^