Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Swallow MSDTC availability exceptions for CI test reliability #77023

Merged
merged 1 commit into from
Oct 18, 2022
Merged

Swallow MSDTC availability exceptions for CI test reliability #77023

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 26 additions & 7 deletions src/libraries/System.Transactions.Local/tests/OleTxTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,10 @@
// The .NET Foundation licenses this file to you under the MIT license.

using System.IO;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
using System.Threading;
using Microsoft.DotNet.RemoteExecutor;
using Xunit;
using Xunit.Abstractions;
using Xunit.Sdk;

namespace System.Transactions.Tests;
Expand Down Expand Up @@ -542,7 +541,7 @@ public void ImplicitDistributedTransactions_cannot_be_changed_after_being_read_a
{
TransactionManager.ImplicitDistributedTransactions = true;

MinimalOleTxScenario();
Test(MinimalOleTxScenario);

Assert.Throws<InvalidOperationException>(() => TransactionManager.ImplicitDistributedTransactions = false);
TransactionManager.ImplicitDistributedTransactions = true;
Expand Down Expand Up @@ -575,6 +574,11 @@ private static void Test(Action action)
return;
}

if (s_isTestSuiteDisabled)
{
return;
}

TransactionManager.ImplicitDistributedTransactions = true;

// In CI, we sometimes get XACT_E_TMNOTAVAILABLE; when it happens, it's typically on the very first
Expand All @@ -589,14 +593,27 @@ private static void Test(Action action)
action();
return;
}
catch (TransactionException e) when (e.InnerException is TransactionManagerCommunicationException)
catch (Exception e) when (e is TransactionManagerCommunicationException or TransactionException { InnerException: TransactionManagerCommunicationException })
{
if (--nRetries == 0)
if (--nRetries > 0)
{
throw;
Thread.Sleep(1000);

continue;
}

Thread.Sleep(1000);
// We've continuously gotten XACT_E_TMNOTAVAILABLE for the entire retry window - MSDTC is unavailable in some way.
// We don't want this to make our CI flaky, so we swallow the exception and skip all subsequent tests.
if (!string.IsNullOrEmpty(Environment.GetEnvironmentVariable("DOTNET_CI")) ||
!string.IsNullOrEmpty(Environment.GetEnvironmentVariable("HELIX_WORKITEM_ROOT")) ||
!string.IsNullOrEmpty(Environment.GetEnvironmentVariable("AGENT_OS")))
{
s_isTestSuiteDisabled = true;

return;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stephentoub do we have a standard way to detect CI, so that this can continue throwing when run locally (e.g. GetEnvironmentVariable("CI") or something)?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not aware of a method we call anywhere to ask whether we're running in CI, but we do have an attribute we apply to suppress a test when in CI:

[Trait(XunitConstants.Category, XunitConstants.IgnoreForCI)]

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I'll take a look at how that's implemented.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the implementation you're looking for:

https://github.com/dotnet/arcade/blob/main/src/Microsoft.DotNet.XUnitExtensions/src/Discoverers/SkipOnCIDiscoverer.cs#L19-L21

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

}

throw;
}
}
}
Expand Down Expand Up @@ -647,4 +664,6 @@ public class OleTxFixture
public OleTxFixture()
=> Test(MinimalOleTxScenario);
}

private static bool s_isTestSuiteDisabled;
}