[Python] Allow users to pass service name for profiler #26220

riteshghorse · 2023-04-11T14:37:26Z

This PR adds an experimental pipeline option for users to pass a service name for profiler. By default, cloud profiler uses JOB_NAME as the service name.
Users can specify this as --dataflow_service_options=enable_google_cloud_profiler=new_profiler

Sample Streaming Wordcount Job: https://pantheon.corp.google.com/dataflow/jobs/us-central1/2023-05-03_07_19_52-5911424696235359677;graphView=0?pageState=(%22dfTime%22:(%22s%22:%222023-05-03T14:19:52.473Z%22,%22e%22:%222023-05-03T14:31:38.548Z%22))&project=google.com:clouddfe

Profiler with the set name:

Fixes #26280

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

…rofiler

riteshghorse · 2023-04-13T20:21:38Z

R: @tvalentyn

github-actions · 2023-04-13T20:22:45Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

sdks/python/apache_beam/runners/worker/sdk_worker_main.py

sdks/python/apache_beam/options/pipeline_options.py

sdks/python/apache_beam/runners/worker/sdk_worker_main.py

codecov · 2023-04-17T17:35:39Z

Codecov Report

Merging #26220 (1706a55) into master (80f1f6c) will increase coverage by 0.01%.
The diff coverage is 68.96%.

@@            Coverage Diff             @@
##           master   #26220      +/-   ##
==========================================
+ Coverage   81.07%   81.08%   +0.01%     
==========================================
  Files         469      469              
  Lines       67165    67199      +34     
==========================================
+ Hits        54453    54488      +35     
+ Misses      12712    12711       -1

Flag	Coverage Δ
python	`81.08% <68.96%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
.../python/apache_beam/runners/worker/data_sampler.py	`100.00% <ø> (ø)`
...thon/apache_beam/runners/worker/sdk_worker_main.py	`77.71% <43.75%> (+0.83%)`	⬆️
...dks/python/apache_beam/options/pipeline_options.py	`94.19% <100.00%> (+0.16%)`	⬆️

... and 20 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

sdks/python/apache_beam/options/pipeline_options.py

sdks/python/apache_beam/options/pipeline_options_test.py

sdks/python/apache_beam/runners/worker/sdk_worker_main.py

CHANGES.md

sdks/python/apache_beam/runners/worker/sdk_worker_main.py

tvalentyn · 2023-04-24T13:36:34Z

thanks, looking better!

tvalentyn · 2023-04-24T15:52:59Z

Thanks, @riteshghorse . We can merge once tests pass, feel free to ping me. Please also re-do the manual sanity check as described in the PR description just to make sure.

tvalentyn · 2023-04-24T18:39:21Z

Run Python_Runners PreCommit

tvalentyn · 2023-04-24T18:39:34Z

Run Python_Integration PreCommit

riteshghorse · 2023-04-24T19:18:49Z

I'm seeing a strange behavior with Dataflow UI. The profiler is generated for this Job. But the Job Info panel on job page doesn't show a job profiler tab as it should. (It shows the tab when I pass dataflow_service_options=enable_google_cloud_profiler but not when dataflow_service_options=enable_google_cloud_profiler=custom is passed)

riteshghorse · 2023-04-24T20:16:46Z

Let's hold merging this until we get response from UI team

riteshghorse · 2023-05-03T14:37:46Z

Changes are live in the UI. Updated the description with new job and profiler example.

Merging this change.

* handle service name for profiler * add debug messages * remove logs * move import statement and logs * rm unnecessary check * add helper to GCloudOption * rm lint * added new helper and unit test * separate functions * updated unit tests * refactored plus check for envvar * updated changes.md

handle service name for profiler

c909a36

github-actions bot added the python label Apr 11, 2023

riteshghorse added 3 commits April 12, 2023 11:17

add debug messages

378bee4

Merge branch 'master' of https://github.com/apache/beam into gcloud-p…

1249777

…rofiler

remove logs

543e53b

riteshghorse marked this pull request as ready for review April 13, 2023 20:08

riteshghorse added 2 commits April 13, 2023 16:14

move import statement and logs

76a5a70

move service version

37fce0e

rm unnecessary check

9430fd2

tvalentyn requested changes Apr 13, 2023

View reviewed changes

riteshghorse added 2 commits April 14, 2023 11:39

add helper to GCloudOption

46f910e

rm lint

901c603

tvalentyn requested changes Apr 14, 2023

View reviewed changes

riteshghorse added 2 commits April 17, 2023 13:09

added new helper and unit test

c9ba755

merge master

74ac6b0

tvalentyn requested changes Apr 18, 2023

View reviewed changes

sdks/python/apache_beam/options/pipeline_options.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/options/pipeline_options_test.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/runners/worker/sdk_worker_main.py Outdated Show resolved Hide resolved

riteshghorse added 2 commits April 21, 2023 11:23

separate functions

149636d

updated unit tests

f4ad1a0

tvalentyn reviewed Apr 24, 2023

View reviewed changes

CHANGES.md Outdated Show resolved Hide resolved

sdks/python/apache_beam/runners/worker/sdk_worker_main.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/runners/worker/sdk_worker_main.py Show resolved Hide resolved

riteshghorse added 2 commits April 24, 2023 11:37

refactored plus check for envvar

bd97905

updated changes.md

1706a55

tvalentyn approved these changes Apr 24, 2023

View reviewed changes

riteshghorse merged commit a13749f into apache:master May 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Python] Allow users to pass service name for profiler #26220

[Python] Allow users to pass service name for profiler #26220

riteshghorse commented Apr 11, 2023 •

edited

Loading

riteshghorse commented Apr 13, 2023

github-actions bot commented Apr 13, 2023

codecov bot commented Apr 17, 2023 •

edited

Loading

tvalentyn commented Apr 24, 2023

tvalentyn commented Apr 24, 2023

tvalentyn commented Apr 24, 2023

tvalentyn commented Apr 24, 2023

riteshghorse commented Apr 24, 2023

riteshghorse commented Apr 24, 2023

riteshghorse commented May 3, 2023

[Python] Allow users to pass service name for profiler #26220

[Python] Allow users to pass service name for profiler #26220

Conversation

riteshghorse commented Apr 11, 2023 • edited Loading

GitHub Actions Tests Status (on master branch)

riteshghorse commented Apr 13, 2023

github-actions bot commented Apr 13, 2023

codecov bot commented Apr 17, 2023 • edited Loading

Codecov Report

tvalentyn commented Apr 24, 2023

tvalentyn commented Apr 24, 2023

tvalentyn commented Apr 24, 2023

tvalentyn commented Apr 24, 2023

riteshghorse commented Apr 24, 2023

riteshghorse commented Apr 24, 2023

riteshghorse commented May 3, 2023

riteshghorse commented Apr 11, 2023 •

edited

Loading

codecov bot commented Apr 17, 2023 •

edited

Loading