Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump Beam SDK to 2.60.0 #2658

Merged
merged 2 commits into from
Oct 22, 2024
Merged

Bump Beam SDK to 2.60.0 #2658

merged 2 commits into from
Oct 22, 2024

Conversation

akkomar
Copy link
Contributor

@akkomar akkomar commented Oct 21, 2024

In version 2.52 Beam removed all Avro-dependent code from core package. I filed https://mozilla-hub.atlassian.net/browse/DENG-2029 last year to get rid of Avro in our codebase because of that. It turned out that BigQueryIO uses Avro, so we need to add beam-sdks-java-extensions-avro dependency anyway. Therefore I'm not removing Avro support in this PR.

Deployment caveat: PubSubIO's read is update incompatible with previous versions so streaming jobs can't be updated.

I tested this on a synthetic structured-logging stream in https://console.cloud.google.com/dataflow/jobs/us-west1/2024-10-21_05_13_30-3138935860679087156;graphView=0?project=akomar-server-telemetry-poc&pageState=(%22dfTime%22:(%22l%22:%22dfJobMaxTime%22)). I haven't seen any issues, but it would probably be good to run this in stage for few hours first.

closes #2651

Downgrade to 2.52.0
@akkomar akkomar requested a review from whd October 21, 2024 12:35
@codecov-commenter
Copy link

codecov-commenter commented Oct 21, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.78%. Comparing base (e90cebc) to head (6346fcd).

Additional details and impacted files
@@            Coverage Diff            @@
##               main    #2658   +/-   ##
=========================================
  Coverage     85.78%   85.78%           
  Complexity      873      873           
=========================================
  Files           118      118           
  Lines          4975     4975           
  Branches        502      502           
=========================================
  Hits           4268     4268           
  Misses          546      546           
  Partials        161      161           
Flag Coverage Δ
ingestion_beam 83.23% <ø> (ø)
ingestion_core 73.64% <ø> (ø)
ingestion_sink 84.07% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@akkomar akkomar mentioned this pull request Oct 22, 2024
Copy link
Member

@whd whd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deployment caveat: PubSubIO's read is update incompatible with previous versions so streaming jobs can't be updated.

Good to know. We currently treat all code changes as non-updateable for this reason.

I'll run this in stage and merge after a few hours. We no longer send a 1% sample of structured and telemetry data there by default but there's at least some data going to stub installer.

@whd whd merged commit 50925a1 into main Oct 22, 2024
10 checks passed
@whd whd deleted the bump_beam branch October 22, 2024 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants