Libraries tests timing out on AlmaLinux

[agocke]

Ever since we onboarded AlmaLinux in #86342 the Libraries Test Run release coreclr linux x64 Release and Libraries Test Run release mono linux x64 Release runs have been timing out. Likely some interaction in the tests themselves.

[ghost]

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Ever since we onboarded AlmaLinux in #86342 the Libraries Test Run release coreclr linux x64 Release and Libraries Test Run release mono linux x64 Release runs have been timing out. Likely some interaction in the tests themselves.

Author:	agocke
Assignees:	-
Labels:	area-Infrastructure-libraries, untriaged
Milestone:	-

[agocke]

This appears to be an issue with MSBuild in this configuration. When I run the System.Diagnostics.Process tests manually, they run and complete successfully, but the MSBuild Exec call never finishes.

I've attached a binlog of the issue.
msbuild.zip

@rainersigwald @dsplaisted Any ideas? Anything I could do to narrow down the hang?

This is in a docker container that was a pretty big pain to setup, so I can help run commands.

[ViktorHofer]

The Libraries Test Run legs don't invoke msbuild. They just execute the RunTests.cmd/sh scripts on the helix clients.

[agocke]

Hmm. Is there a good way to mimic the behavior of running all the tests locally? So I can figure out which one is failing?

[agocke]

Or to get this info from Helix? All I see in the helix log is this: https://helix.dot.net/api/jobs/6eeb06e8-f3e5-4897-bd0b-70e966604916/details?api-version=2019-06-17

[ViktorHofer]

Unfortunately, I don't know the answer to that. I would defer to MattGal or Santi but they aren't part of the team anymore. What I know is that there isn't a way to reproduce how tests are run on Helix, locally - aside from manually executing the RunTests scripts after the test assemblies are built (libs.tests subset).

[ViktorHofer]

@agocke have you had time to reproduce the timeouts locally on an AlmaLinux container?

[agocke]

I tried, no luck. My initial results where that it looked like both System.Diagnostics.Process and something else (sorry, left on another computer) were timing out.

[ViktorHofer]

There's not really much that we can do at this point without spending a ton of time on this. Moving to Future.