[Beam#25046]add support for flink 1.16 for Beam Flink Runner #25584

yananhao12 · 2023-02-21T23:27:57Z

#25046: add support for flink 1.16 for Beam Flink Runner

Incompatibility fixed: use TestBaseUtils.compareResultsByLinesInMemory directly in tests since AbstractTestBase no longer inherits TestBaseUtils

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

yananhao12 · 2023-02-21T23:58:58Z

@xinyuiscool

github-actions · 2023-02-22T00:15:21Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @damccorm for label python.
R: @Abacn for label build.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

codecov · 2023-02-22T01:09:38Z

Codecov Report

Merging #25584 (8675728) into master (948f7cd) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master   #25584      +/-   ##
==========================================
- Coverage   72.80%   72.79%   -0.01%     
==========================================
  Files         775      775              
  Lines      102866   102866              
==========================================
- Hits        74887    74877      -10     
- Misses      26524    26534      +10     
  Partials     1455     1455

Flag	Coverage Δ
python	`81.93% <100.00%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...dks/python/apache_beam/options/pipeline_options.py	`93.97% <100.00%> (ø)`
.../beam/runners/prism/internal/jobservices/server.go	`68.00% <0.00%> (-8.00%)`	⬇️
.../python/apache_beam/testing/test_stream_service.py	`88.09% <0.00%> (-4.77%)`	⬇️
...python/apache_beam/runners/worker/worker_status.py	`75.33% <0.00%> (-1.34%)`	⬇️
...che_beam/runners/interactive/interactive_runner.py	`90.50% <0.00%> (-1.27%)`	⬇️
...eam/runners/portability/fn_api_runner/execution.py	`92.49% <0.00%> (-0.64%)`	⬇️
...on/apache_beam/runners/dataflow/dataflow_runner.py	`81.74% <0.00%> (-0.15%)`	⬇️
...hon/apache_beam/runners/direct/test_stream_impl.py	`94.02% <0.00%> (+0.74%)`	⬆️
sdks/python/apache_beam/io/localfilesystem.py	`91.72% <0.00%> (+0.75%)`	⬆️
sdks/python/apache_beam/internal/metrics/metric.py	`94.00% <0.00%> (+1.00%)`	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

becketqin · 2023-02-22T01:54:46Z

gradle.properties

@@ -38,5 +38,5 @@ javaVersion=1.8
 docker_image_default_repo_root=apache
 docker_image_default_repo_prefix=beam_

-flink_versions=1.12,1.13,1.14,1.15
+flink_versions=1.12,1.13,1.14,1.15,1.16


I am wondering what is the current strategy of supported Flink versions. Are we only keeping 4 versions at the moment? Ping @mxm @angoenka

By looking at pr history, adding and removing flink versions happen in different prs. So for this pr, we can add the support for 1.16. In the future we can remove 1.12 support.

Last time this was discussed ~2y ago the idea was to maintain the supported versions by Flink (1.16, 1.15) + one older version (1.14). So we should definitely get rid of 1.12 and probably 1.13. Of course an email to the dev@ mailing list would be a good idea, and maybe to put a warning in advance for the next release.

runners/flink/1.16/src/test/java/org/apache/beam/runners/flink/MiniClusterCompat.java

runners/flink/1.16/build.gradle

xinyuiscool

LGTM!

xinyuiscool · 2023-02-22T18:27:05Z

@iemejia : hey, Ismael, do you have time to take a quick look at this pr? It's for upgrading to Flink 1.16. Thanks.

Abacn · 2023-02-22T18:27:34Z

Run Java_PVR_Flink_Batch PreCommit

yananhao12 · 2023-02-22T19:00:39Z

Run Java_Kafka_IO_Direct PreCommit

yananhao12 · 2023-02-23T21:29:44Z

Run Java Tests

becketqin

LGTM.

Abacn · 2023-02-24T02:53:37Z

unit test not running. We need to add two lines here:

beam/build.gradle.kts

Line 244 in dcfaddd

dependsOn(":runners:flink:1.15:job-server:build")

Abacn

test passed, thanks, LGTM

…25584) (cherry picked from commit de71d13)

* [Beam#25046]add support for flink 1.16 for Beam Flink Runner (apache#25584) (cherry picked from commit de71d13) * bump Beam version * fix spotless check --------- Co-authored-by: Yanan Hao <[email protected]>

iemejia · 2023-03-06T13:55:50Z

Sorry I was OOO and missed this one. Looks right, thanks @yananhao12 !
If you decide to go ahead and add the PRs to remove 1.12 and 1.13 feel free to ping me for the review.

…25584)

streamnsight · 2023-03-15T23:09:37Z

great news... now when can we expect the maven repo to be updated? the docker images to run beam on Flink try to load the jars from there but they are not found.

INFO:apache_beam.utils.subprocess_server:Downloading job server jar from https://repo.maven.apache.org/maven2/org/apache/beam/beam-runners-flink-1.16-job-server/2.47.0/beam-runners-flink-1.16-job-server-2.47.0.jar not found

Abacn · 2023-03-15T23:12:04Z

@streamnsight This will be shipped with Beam 2.47.0 which has not been released. Scheduled late April.

Abacn · 2023-03-16T14:35:18Z

@streamnsight are you running beam at development branch (master)? For released version (2.46.0) it should use flink 1.15 by default; I am confused that the message shows the version is 2.47.0 which does not exist yet (not 2.47.0-SNAPSHOT)

…25584)

add support for flink 1.16

749d1ff

github-actions bot added build flink python runners labels Feb 21, 2023

github-actions bot added the Next Action: Reviewers label Feb 22, 2023

fix splotless check

53743dd

becketqin suggested changes Feb 22, 2023

View reviewed changes

remove unncessary MiniClusterCompat

e17f038

xinyuiscool approved these changes Feb 22, 2023

View reviewed changes

xinyuiscool requested a review from iemejia February 22, 2023 18:24

Merge branch 'apache:master' into flink_1.16

5691efc

Merge branch 'apache:master' into flink_1.16

a8ed800

becketqin approved these changes Feb 24, 2023

View reviewed changes

register flink 1.16 in precommit and postcommit test

8675728

Abacn approved these changes Feb 24, 2023

View reviewed changes

xinyuiscool merged commit de71d13 into apache:master Feb 24, 2023

Abacn mentioned this pull request Feb 24, 2023

Update CHANGE.md to 2.47.0 #25628

Merged

3 tasks

yananhao12 deleted the flink_1.16 branch February 24, 2023 18:55

yananhao12 mentioned this pull request Feb 24, 2023

[CODA-161] Add support for Flink 1.16 linkedin/beam#79

Closed

3 tasks

yananhao12 added a commit to yananhao12/beam that referenced this pull request Feb 28, 2023

[Beam#25046]add support for flink 1.16 for Beam Flink Runner (apache#…

ee4b81c

…25584) (cherry picked from commit de71d13)

yananhao12 mentioned this pull request Feb 28, 2023

[cherry-pick] add support for flink 1.16 for Beam Flink Runner linkedin/beam#80

Merged

3 tasks

ruslan-ikhsan pushed a commit to akvelon/beam that referenced this pull request Mar 10, 2023

[Beam#25046]add support for flink 1.16 for Beam Flink Runner (apache#…

bc0ba25

…25584)

steveniemitz pushed a commit to twitter-forks/beam that referenced this pull request May 10, 2023

[Beam#25046]add support for flink 1.16 for Beam Flink Runner (apache#…

2735d24

…25584)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Beam#25046]add support for flink 1.16 for Beam Flink Runner #25584

[Beam#25046]add support for flink 1.16 for Beam Flink Runner #25584

yananhao12 commented Feb 21, 2023 •

edited

Loading

yananhao12 commented Feb 21, 2023

github-actions bot commented Feb 22, 2023

codecov bot commented Feb 22, 2023 •

edited

Loading

becketqin Feb 22, 2023

xinyuiscool Feb 22, 2023

iemejia Mar 6, 2023

xinyuiscool left a comment

xinyuiscool commented Feb 22, 2023

Abacn commented Feb 22, 2023

yananhao12 commented Feb 22, 2023

yananhao12 commented Feb 23, 2023

becketqin left a comment

Abacn commented Feb 24, 2023

Abacn left a comment

iemejia commented Mar 6, 2023

streamnsight commented Mar 15, 2023

Abacn commented Mar 15, 2023

Abacn commented Mar 16, 2023

[Beam#25046]add support for flink 1.16 for Beam Flink Runner #25584

[Beam#25046]add support for flink 1.16 for Beam Flink Runner #25584

Conversation

yananhao12 commented Feb 21, 2023 • edited Loading

GitHub Actions Tests Status (on master branch)

yananhao12 commented Feb 21, 2023

github-actions bot commented Feb 22, 2023

codecov bot commented Feb 22, 2023 • edited Loading

Codecov Report

becketqin Feb 22, 2023

Choose a reason for hiding this comment

xinyuiscool Feb 22, 2023

Choose a reason for hiding this comment

iemejia Mar 6, 2023

Choose a reason for hiding this comment

xinyuiscool left a comment

Choose a reason for hiding this comment

xinyuiscool commented Feb 22, 2023

Abacn commented Feb 22, 2023

yananhao12 commented Feb 22, 2023

yananhao12 commented Feb 23, 2023

becketqin left a comment

Choose a reason for hiding this comment

Abacn commented Feb 24, 2023

Abacn left a comment

Choose a reason for hiding this comment

iemejia commented Mar 6, 2023

streamnsight commented Mar 15, 2023

Abacn commented Mar 15, 2023

Abacn commented Mar 16, 2023

yananhao12 commented Feb 21, 2023 •

edited

Loading

codecov bot commented Feb 22, 2023 •

edited

Loading