Don't wait for finalizers in 'IReferenceTrackerHost::ReleaseDisconnec…

…tedReferenceSources' (#110551)

This PR updates the IReferenceTrackerHost::ReleaseDisconnectedReferenceSources implementation for CoreCLR and NativeAOT to match what .NET Native was doing, and not wait for finalizers, to avoid deadlocks in ASTA scenarios (UWP). Finalizers will just continue running normally, and if the process is suspended (which can only happen on UWP anyway), they'll resume when the process is resumed later on. Worst case scenario this would only cause a non-optimal memory use cleanup before suspension (which is better than a deadlock anyway). Can be revisited and improved for .NET 10 if needed.

Contributes to #109538

src/coreclr/interop/trackerobjectmanager.cpp

@@ -84,7 +84,10 @@ namespace
 
     STDMETHODIMP HostServices::ReleaseDisconnectedReferenceSources()
     {
-        return InteropLibImports::WaitForRuntimeFinalizerForExternal();
+        // We'd like to call InteropLibImports::WaitForRuntimeFinalizerForExternal() here, but this could
+        // lead to deadlock if the finalizer thread is trying to get back to this thread, because we are
+        // not pumping anymore. Disable this for now. See: https://github.com/dotnet/runtime/issues/109538.
+        return S_OK;
     }
 
     STDMETHODIMP HostServices::NotifyEndOfReferenceTrackingOnThread()

src/coreclr/nativeaot/System.Private.CoreLib/src/System/Runtime/InteropServices/ComWrappers.NativeAot.cs

@@ -1418,15 +1418,10 @@ internal static unsafe int IReferenceTrackerHost_DisconnectUnusedReferenceSource
         [UnmanagedCallersOnly]
         internal static unsafe int IReferenceTrackerHost_ReleaseDisconnectedReferenceSources(IntPtr pThis)
         {
-            try
-            {
-                GC.WaitForPendingFinalizers();
-                return HResults.S_OK;
-            }
-            catch (Exception e)
-            {
-                return Marshal.GetHRForException(e);
-            }
+            // We'd like to call GC.WaitForPendingFinalizers() here, but this could lead to deadlock
+            // if the finalizer thread is trying to get back to this thread, because we are not pumping
+            // anymore. Disable this for now. See: https://github.com/dotnet/runtime/issues/109538.
+            return HResults.S_OK;
         }
 
         [UnmanagedCallersOnly]