Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gh-94182: run the PidfdChildWatcher on the running loop #94184

Merged
merged 33 commits into from
Oct 8, 2022
Merged

gh-94182: run the PidfdChildWatcher on the running loop #94184

merged 33 commits into from
Oct 8, 2022

Conversation

graingert
Copy link
Contributor

@graingert graingert commented Jun 23, 2022 •
edited by bedevere-bot
Loading

graingert
graingert commented Jun 23, 2022
View reviewed changes
Lib/test/test_asyncio/test_subprocess.py Outdated
@@ -712,7 +712,7 @@ def setUp(self):
self.set_event_loop(self.loop)


class GenericWatcherTests:
class GenericWatcherTests(test_utils.TestCase):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this test wasn't enabled?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I split this fix out, so this PR is easier to review: #95009

graingert
graingert commented Jun 24, 2022
View reviewed changes
Lib/test/test_asyncio/test_subprocess.py Outdated

async def execute():
watcher = mock.create_authspec(asyncio.AbstractChildWatcher)
watcher.is_active.return_value = False
Copy link
Contributor Author

@graingert graingert Jun 24, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was patching the wrong attribute - asyncio calls asyncio.get_child_watcher().__enter__().is_active()

graingert and others added 2 commits June 24, 2022 09:10
@graingert
check that asyncio.set_event_loop() was not called
6e56155 
otherwise it invalidates test_create_subprocess_with_pidfd
@blurb-it
📜🤖 Added by blurb_it.
1037078
@graingert graingert marked this pull request as ready for review June 24, 2022 08:50
@graingert graingert requested review from 1st1 and asvetlov as code owners June 24, 2022 08:50
@graingert
Copy link
Contributor Author

@asvetlov can I get a review on this please?

@gvanrossum gvanrossum self-requested a review as a code owner October 7, 2022 18:55
gvanrossum
gvanrossum approved these changes Oct 7, 2022
View reviewed changes
Copy link
Member

@gvanrossum gvanrossum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. We'll probably merge this ASAP.

Lib/test/test_asyncio/test_subprocess.py
@@ -30,6 +31,19 @@
'sys.stdout.buffer.write(data)'))]


@functools.cache
def _has_pidfd_support():
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This same function is also introduced by @kumaraditya303's PR GH-98024 (which uses PidfdChildWatcher when supported). I propose to refactor things so that everything uses Kumar's version. We can do that in Kumar's PR once this PR is merged.

Lib/asyncio/unix_events.py
def _do_wait(self, pid):
pidfd, callback, args = self._callbacks.pop(pid)
self._loop._remove_reader(pidfd)
def _do_wait(self, pid, pidfd, callback, args):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious why we can't pass the loop as an argument as well? (Though maybe that creates an unhealthy self-reference to the loop.)

@gvanrossum
Copy link
Member

(Waiting for @1st1 review and passing tests.)

@gvanrossum gvanrossum mentioned this pull request Oct 7, 2022
Merged
@gvanrossum
Copy link
Member

Not sure what's wrong with the docs build.

@1st1
Copy link
Member

1st1 commented Oct 7, 2022

Not sure what's wrong with the docs build.

I've fixed it.

@1st1
Copy link
Member

1st1 commented Oct 7, 2022

I've fixed it.

Reverted the "fix" because apparently it was caused by an unrelated commit in main.

@JelleZijlstra JelleZijlstra self-assigned this Oct 7, 2022
@JelleZijlstra
Copy link
Member

I can take care of getting CI green and hitting the merge button here.

@1st1
Copy link
Member

1st1 commented Oct 7, 2022

I can take care of getting CI green and hitting the merge button here.

Please do. You might need to rebase this PR to the latest main.

@gvanrossum
Copy link
Member

The macOS build hit some GitHub Actions limit. I'm going to merge without waiting for it.

@gvanrossum gvanrossum merged commit 3d8b224 into python:main Oct 8, 2022
carljm added a commit to carljm/cpython that referenced this pull request Oct 8, 2022
@carljm
Merge branch 'main' into typewatch
d67d6fd 
* main:
  pythongh-68686: Retire eptag ptag scripts (python#98064)
  pythongh-97922: Run the GC only on eval breaker (python#97920)
  GitHub Workflows security hardening (python#96492)
  Add `@ezio-melotti` as codeowner for `.github/`. (python#98079)
  pythongh-97913 Docs: Add walrus operator to the index (python#97921)
  [doc] Fix broken links to C extensions accelerating stdlib modules (python#96914)
  pythongh-97822: Fix http.server documentation reference to test() function (python#98027)
  pythongh-91052: Add PyDict_Unwatch for unwatching a dictionary (python#98055)
  pythonGH-98023: Change default child watcher to PidfdChildWatcher on supported systems (python#98024)
  pythonGH-94182: Run the PidfdChildWatcher on the running loop (python#94184)
carljm added a commit to carljm/cpython that referenced this pull request Oct 9, 2022
@carljm
Merge branch 'main' into urllibnonet
132c32e 
* main: (5519 commits)
  Minor edits to the Descriptor HowTo Guide (pythonGH-24901)
  Fix link to Lifecycle of a Pull Request in CONTRIBUTING (python#98102)
  pythonGH-94597: deprecate `SafeChildWatcher`, `FastChildWatcher` and `MultiLoopChildWatcher` child watchers  (python#98089)
  Auto-cancel old builds when new commit pushed to branch (python#98009)
  pythongh-95011: Migrate syslog module to Argument Clinic (pythonGH-95012)
  pythongh-68686: Retire eptag ptag scripts (python#98064)
  pythongh-97922: Run the GC only on eval breaker (python#97920)
  GitHub Workflows security hardening (python#96492)
  Add `@ezio-melotti` as codeowner for `.github/`. (python#98079)
  pythongh-97913 Docs: Add walrus operator to the index (python#97921)
  [doc] Fix broken links to C extensions accelerating stdlib modules (python#96914)
  pythongh-97822: Fix http.server documentation reference to test() function (python#98027)
  pythongh-91052: Add PyDict_Unwatch for unwatching a dictionary (python#98055)
  pythonGH-98023: Change default child watcher to PidfdChildWatcher on supported systems (python#98024)
  pythonGH-94182: Run the PidfdChildWatcher on the running loop (python#94184)
  pythongh-92886: make test_ast pass with -O (assertions off) (pythonGH-98058)
  pythongh-92886: make test_coroutines pass with -O (assertions off) (pythonGH-98060)
  pythongh-57179: Add note on symlinks for os.walk (python#94799)
  pythongh-94808: Fix regex on exotic platforms (python#98036)
  pythongh-90085: Remove vestigial -t and -c timeit options (python#94941)
  ...
mpage pushed a commit to mpage/cpython that referenced this pull request Oct 11, 2022
@graingert @1st1
@JelleZijlstra @mpage
pythonGH-94182: Run the PidfdChildWatcher on the running loop (python…
6cd489d 
…#94184)

There is no reason for this watcher to be attached to any particular loop.
This should make it safe to use regardless of the lifetime of the event loop running in the main thread
(relative to other loops).

Co-authored-by: Yury Selivanov <[email protected]>
Co-authored-by: Jelle Zijlstra <[email protected]>
bdraco added a commit to bdraco/home-assistant that referenced this pull request Feb 12, 2023
@bdraco
Use PidfdChildWatcher by default when available
29f4931 
This is a backport from cpython 3.12

https://docs.python.org/3/library/asyncio-policy.html
> PidfdChildWatcher is a “Goldilocks” child watcher implementation. It doesn’t require signals or threads, doesn’t interfere with any processes launched outside the event loop, and scales linearly with the number of subprocesses launched by the event loop. The main disadvantage is that pidfds are specific to Linux, and only work on recent (5.3+) kernels.

python/cpython#98024
There are some additional fixes in cpython 3.12
in python/cpython#94184
when there is no event loop running in the main thread
but this is not a problem we have
@bdraco bdraco mentioned this pull request Feb 12, 2023
Merged
19 tasks
balloob pushed a commit to home-assistant/core that referenced this pull request Feb 12, 2023
@bdraco
Use PidfdChildWatcher by default when available (#87951)
d1e1734 
This is a backport from cpython 3.12

https://docs.python.org/3/library/asyncio-policy.html
> PidfdChildWatcher is a “Goldilocks” child watcher implementation. It doesn’t require signals or threads, doesn’t interfere with any processes launched outside the event loop, and scales linearly with the number of subprocesses launched by the event loop. The main disadvantage is that pidfds are specific to Linux, and only work on recent (5.3+) kernels.

python/cpython#98024
There are some additional fixes in cpython 3.12
in python/cpython#94184
when there is no event loop running in the main thread
but this is not a problem we have
AlePerla pushed a commit to AlePerla/homeassistant_core that referenced this pull request Feb 17, 2023
@bdraco @AlePerla
Use PidfdChildWatcher by default when available (home-assistant#87951)
638c431 
This is a backport from cpython 3.12

https://docs.python.org/3/library/asyncio-policy.html
> PidfdChildWatcher is a “Goldilocks” child watcher implementation. It doesn’t require signals or threads, doesn’t interfere with any processes launched outside the event loop, and scales linearly with the number of subprocesses launched by the event loop. The main disadvantage is that pidfds are specific to Linux, and only work on recent (5.3+) kernels.

python/cpython#98024
There are some additional fixes in cpython 3.12
in python/cpython#94184
when there is no event loop running in the main thread
but this is not a problem we have
@graingert graingert deleted the run-pidfdchildwatcher-on-the-running-loop branch October 11, 2023 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@1st1 1st1 1st1 approved these changes

@agronholm agronholm agronholm requested changes

@gvanrossum gvanrossum gvanrossum approved these changes

@asvetlov asvetlov Awaiting requested review from asvetlov asvetlov is a code owner

Assignees

@JelleZijlstra JelleZijlstra

Labels
3.10 only security fixes 3.11 only security fixes 3.12 bugs and security fixes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@graingert @gvanrossum @1st1 @JelleZijlstra @agronholm @bedevere-bot