Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EventPipe shutdown race condition in .net7/.net6 #89621

Closed
lateralusX opened this issue Jul 28, 2023 · 1 comment
Closed

EventPipe shutdown race condition in .net7/.net6 #89621

lateralusX opened this issue Jul 28, 2023 · 1 comment

Comments

@lateralusX
Copy link
Member

Description

The fix for f39e05a added a condition to close down the listener's IPC channel during shutdown. Turns out that this can cause a race with the diagnostic server thread if its in the process of using that handle as part of a GetOverlappedResult call that in turn will end up closing the handle:

CloseHandle (ipc->pipe);

This probably will go unnoticed on release builds, but on debug build it could trigger asserts due to internal heap corruption or attempt to close and invalid handle if the race gets exposed.

Currently EventPipe doesn't have any coordination mechanism during shutdown to make sure diagnostic server thread stops during shutdown, so it can't close any of the resources since they might still be used.

Fix is to revert

if (is_shutdown && ipc->mode != DS_IPC_CONNECTION_MODE_LISTEN) {
to keep ipc->pipe around during shutdown:

@@ -634,13 +634,11 @@ ds_ipc_close (
	ds_ipc_error_callback_func callback)
{
	EP_ASSERT (ipc != NULL);

	// don't attempt cleanup on shutdown and let the OS handle it
-    // except in the case of listen pipes - if they leak the process
-    // will fail to reinitialize the pipe for embedding scenarios.
-	if (is_shutdown && ipc->mode != DS_IPC_CONNECTION_MODE_LISTEN) {
+	if (is_shutdown) {
		if (callback)
			callback ("Closing without cleaning underlying handles", 100);
		return;
	}

Reproduction Steps

Running EventPipe test like src\tests\tracing\eventpipe\rundownvalidation using debug build running under debugger.

Expected behavior

No race.

Actual behavior

If race is hit, process will crash.

Regression?

Yes

Known Workarounds

No response

Configuration

No response

Other information

No response

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jul 28, 2023
@lateralusX
Copy link
Member Author

Fixes:

#89619
#89620

@steveisok steveisok removed the untriaged New issue has not been triaged by the area owner label Jul 31, 2023
@hoyosjs hoyosjs added this to the 9.0.0 milestone Jul 31, 2023
@hoyosjs hoyosjs closed this as completed Jul 16, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Aug 16, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants