Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HttpListener tests timing out on Mono in CI #2391

Open
stephentoub opened this issue Jan 30, 2020 · 14 comments
Open

HttpListener tests timing out on Mono in CI #2391

stephentoub opened this issue Jan 30, 2020 · 14 comments
Labels
area-System.Net.Http disabled-test The test is disabled in source code against the issue runtime-mono specific to the Mono runtime
Milestone
Future

Comments

@stephentoub
Copy link
Member

https://helix.dot.net/api/2019-06-17/jobs/50e7502e-a749-4812-a7f0-e5edf3eeba61/workitems/System.Net.HttpListener.Tests/console

  Discovering: System.Net.HttpListener.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.HttpListener.Tests (found 198 of 298 test cases)
  Starting:    System.Net.HttpListener.Tests (parallel test collections = on, max threads = 2)
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.DrainEntityBody_Roundtrips [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.UnsupportedProperties_Throw [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.IdleConnection_Roundtrips [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.Properties_DefaultValues [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.Dispose_CallAfterDisposed_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.ReceiveAsync_Disposed_ThrowsObjectDisposedException [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_AfterDisposed_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseOutputAsync_HandshakeStartedFromClient_Success [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_AfterAborted_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_HandshakeStartedFromClient_Success [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.SendAsync_Disposed_ThrowsObjectDisposedException [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpRequestStreamTests.Read_FromClosedConnectionAsynchronously_ThrowsHttpListenerException', Elapsed: 00:02:04
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpRequestStreamTests.Read_FromClosedConnectionAsynchronously_ThrowsHttpListenerException', Elapsed: 00:04:04
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpRequestStreamTests.Read_FromClosedConnectionAsynchronously_ThrowsHttpListenerException', Elapsed: 00:06:04
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpRequestStreamTests.Read_FromClosedConnectionAsynchronously_ThrowsHttpListenerException', Elapsed: 00:08:04
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpRequestStreamTests.Read_FromClosedConnectionAsynchronously_ThrowsHttpListenerException', Elapsed: 00:10:04
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpRequestStreamTests.Read_FromClosedConnectionAsynchronously_ThrowsHttpListenerException', Elapsed: 00:12:04

cc: @akoeplinger
@akoeplinger
Copy link
Member

I've tried reproducing these System.Net.HttpListener.Tests timeouts but wasn't able to, even using the Docker containers on my Mac.

@directhex can you please try to repro and/or disable these tests?

@stephentoub stephentoub changed the title Read_FromClosedConnectionAsynchronously_ThrowsHttpListenerException test timing out on Mono in CI HttpListener tests timing out on Mono in CI Jan 30, 2020
@stephentoub
Copy link
Member Author

stephentoub commented Jan 30, 2020 •
edited
Loading

Another timeout, different PR, different test:
https://helix.dot.net/api/2019-06-17/jobs/01ad5712-0a89-4684-9e9d-b80612973f2e/workitems/System.Net.HttpListener.Tests/console

  Discovering: System.Net.HttpListener.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.HttpListener.Tests (found 198 of 298 test cases)
  Starting:    System.Net.HttpListener.Tests (parallel test collections = on, max threads = 2)
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.DrainEntityBody_Roundtrips [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.UnsupportedProperties_Throw [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.IdleConnection_Roundtrips [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.Properties_DefaultValues [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.Dispose_CallAfterDisposed_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.ReceiveAsync_Disposed_ThrowsObjectDisposedException [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_AfterDisposed_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseOutputAsync_HandshakeStartedFromClient_Success [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_AfterAborted_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_HandshakeStartedFromClient_Success [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.SendAsync_Disposed_ThrowsObjectDisposedException [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.SendAsync_NoInnerBuffer_ThrowsArgumentNullException', Elapsed: 00:02:05
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.SendAsync_NoInnerBuffer_ThrowsArgumentNullException', Elapsed: 00:04:05
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.SendAsync_NoInnerBuffer_ThrowsArgumentNullException', Elapsed: 00:06:05
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.SendAsync_NoInnerBuffer_ThrowsArgumentNullException', Elapsed: 00:08:05
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.SendAsync_NoInnerBuffer_ThrowsArgumentNullException', Elapsed: 00:10:05
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.SendAsync_NoInnerBuffer_ThrowsArgumentNullException', Elapsed: 00:12:05

@directhex
Copy link
Contributor

@akoeplinger what's the quickest way to try and repro?

@stephentoub
Copy link
Member Author

Another, again different PR, different test:
https://helix.dot.net/api/2019-06-17/jobs/5a44346f-0d12-4dc5-9847-8e3ddda83ed6/workitems/System.Net.HttpListener.Tests/console

  Discovering: System.Net.HttpListener.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.HttpListener.Tests (found 198 of 298 test cases)
  Starting:    System.Net.HttpListener.Tests (parallel test collections = on, max threads = 2)
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.DrainEntityBody_Roundtrips [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.UnsupportedProperties_Throw [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.IdleConnection_Roundtrips [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.Properties_DefaultValues [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.Dispose_CallAfterDisposed_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.ReceiveAsync_Disposed_ThrowsObjectDisposedException [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_AfterDisposed_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseOutputAsync_HandshakeStartedFromClient_Success [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_AfterAborted_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_HandshakeStartedFromClient_Success [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.SendAsync_Disposed_ThrowsObjectDisposedException [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpResponseStreamTests.Write_TooLittleSynchronouslyAndClose_ThrowsInvalidOperationException [SKIP]
      Condition(s) not met: "IsWindowsImplementation"
    System.Net.Tests.HttpResponseStreamTests.Write_TooLittleAsynchronouslyAndClose_ThrowsInvalidOperationException [SKIP]
      Condition(s) not met: "IsWindowsImplementation"
    System.Net.Tests.HttpResponseStreamTests.Write_TooMuch_ThrowsProtocolViolationException [SKIP]
      Condition(s) not met: "IsWindowsImplementation"
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpResponseStreamTests.EndWrite_CalledTwice_ThrowsInvalidOperationException', Elapsed: 00:02:03
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpResponseStreamTests.EndWrite_CalledTwice_ThrowsInvalidOperationException', Elapsed: 00:04:03
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpResponseStreamTests.EndWrite_CalledTwice_ThrowsInvalidOperationException', Elapsed: 00:06:03
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpResponseStreamTests.EndWrite_CalledTwice_ThrowsInvalidOperationException', Elapsed: 00:08:03
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpResponseStreamTests.EndWrite_CalledTwice_ThrowsInvalidOperationException', Elapsed: 00:10:03
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpResponseStreamTests.EndWrite_CalledTwice_ThrowsInvalidOperationException', Elapsed: 00:12:03

@akoeplinger
Copy link
Member

@directhex The way I did was to look at e.g. https://helix.dot.net/api/2019-06-17/jobs/01ad5712-0a89-4684-9e9d-b80612973f2e/workitems/System.Net.HttpListener.Tests and download the run_client.py log which contains the various payloads, then I launched a docker container, downloaded and unzipped all the payloads and used the RunTests.sh from there.

@directhex
Copy link
Contributor

Starting: System.Net.HttpListener.Tests (parallel test collections = on, max threads = 8)

@directhex
Copy link
Contributor

Which Docker image do we use to run this on Helix? I want my runs to match where it's failing as closely as possible

@stephentoub
Copy link
Member Author

Another failure here, though not manifesting in the same way (so not sure if it's actually the same or not):
https://helix.dot.net/api/2019-06-17/jobs/b9a45ad2-d9fb-455a-a6e6-e66dc29cf73b/workitems/System.Net.HttpListener.Tests/console

    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_InvalidCloseStatus_ThrowsArgumentException(status: EndpointUnavailable, statusDescription: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"..., paramName: "statusDescription") [FAIL]
      System.Net.WebSockets.WebSocketException : The server returned status code '400' when status code '101' was expected.
      Stack Trace:
           at System.Net.WebSockets.WebSocketHandle.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options)
           at System.Net.WebSockets.ClientWebSocket.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken)
           at System.Net.Tests.HttpListenerWebSocketTests.GetWebSocketContext(String[] subProtocols)
           at System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_InvalidCloseStatus_ThrowsArgumentException(WebSocketCloseStatus status, String statusDescription, String paramName)
        --- End of stack trace from previous location ---

@directhex
Copy link
Contributor

I've just run these tests 40 times without reproducing the problem 🤷‍♂️

@danmoseley
Copy link
Member

What configuration did you use @directhex ?

Eg., I've seen it on netcoreapp5.0-Linux-Debug-x64-Mono_release-Centos.7.Amd64.Open

@safern can tell you the docker image.

@danmoseley
Copy link
Member

https://dev.azure.com/dnceng/public/_build/results?buildId=502018&view=ms.vss-test-web.build-test-results-tab&runId=15945120&resultId=168480&paneView=debug is another instance, it's on
netcoreapp5.0-Linux-Debug-x64-Mono_release-(Fedora.29.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:fedora-29-helix-a12566d-20191210224553

@safern
Copy link
Member

safern commented Jan 30, 2020

I just looked at all the links that @stephentoub pasted, none of them ran in a docker container, I found that there are 3 different flavors that hit this issue:

Centos7
Debian9
SLES15

Maybe that's why you can't repro?

This is the test dll and its dependency for one of the jobs above:
https://helixde107v0xdeko0k025g8.blob.core.windows.net/helix-job-4c5b489e-62be-4fc0-8eaa-319efce3de0d30db8ad0257421b8f/System.Net.HttpListener.Tests.zip?sv=2019-02-02&se=2020-02-09T15%3A10%3A50Z&sr=c&sp=rl&sig=KC2ck%2BrOaDkIC7SJoLu%2BtY4tqbwpUsnrqJju9t34%2BOQ%3D

And this is the correlation payload (testhost)
https://helixde107v0xdeko0k025g8.blob.core.windows.net/helix-job-4c5b489e-62be-4fc0-8eaa-319efce3de0d30db8ad0257421b8f/test-runtime-netcoreapp5.0-Linux-Debug-x64.zip?sv=2019-02-02&se=2020-02-09T15%3A10%3A50Z&sr=c&sp=rl&sig=KC2ck%2BrOaDkIC7SJoLu%2BtY4tqbwpUsnrqJju9t34%2BOQ%3D

That one ran on:
sles.15.amd64.open

@stephentoub
Copy link
Member Author

https://helix.dot.net/api/2019-06-17/jobs/d8881f6f-0bc3-4c80-8400-d1e445618ad8/workitems/System.Net.HttpListener.Tests/console

  Discovering: System.Net.HttpListener.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.HttpListener.Tests (found 198 of 298 test cases)
  Starting:    System.Net.HttpListener.Tests (parallel test collections = on, max threads = 2)
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.DrainEntityBody_Roundtrips [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.UnsupportedProperties_Throw [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.IdleConnection_Roundtrips [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.Properties_DefaultValues [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.Dispose_CallAfterDisposed_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.ReceiveAsync_Disposed_ThrowsObjectDisposedException [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_AfterDisposed_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseOutputAsync_HandshakeStartedFromClient_Success [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_AfterAborted_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_HandshakeStartedFromClient_Success [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.SendAsync_Disposed_ThrowsObjectDisposedException [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_InvalidCloseStatus_ThrowsArgumentException', Elapsed: 00:02:04
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_InvalidCloseStatus_ThrowsArgumentException', Elapsed: 00:04:04
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_InvalidCloseStatus_ThrowsArgumentException', Elapsed: 00:06:04
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_InvalidCloseStatus_ThrowsArgumentException', Elapsed: 00:08:04
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_InvalidCloseStatus_ThrowsArgumentException', Elapsed: 00:10:04
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_InvalidCloseStatus_ThrowsArgumentException', Elapsed: 00:12:04

@stephentoub stephentoub mentioned this issue Jan 31, 2020
Merged
@stephentoub stephentoub added the disabled-test The test is disabled in source code against the issue label Jan 31, 2020
@stephentoub stephentoub added this to the 5.0 milestone Jan 31, 2020
@akoeplinger akoeplinger mentioned this issue Jan 31, 2020
Closed
@stephentoub stephentoub added the runtime-mono specific to the Mono runtime label Feb 25, 2020
@karelz karelz modified the milestones: 5.0, Future Jun 5, 2020
@dotnet-policy-service Dotnet Policy Service
Copy link
Contributor

Due to lack of recent activity, this issue has been marked as a candidate for backlog cleanup. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will undo this process.

This process is part of our issue cleanup automation.

@dotnet-policy-service dotnet-policy-service bot added backlog-cleanup-candidate An inactive issue that has been marked for automated closure. no-recent-activity labels Dec 5, 2024
@MihaZupan MihaZupan removed no-recent-activity backlog-cleanup-candidate An inactive issue that has been marked for automated closure. labels Dec 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area-System.Net.Http disabled-test The test is disabled in source code against the issue runtime-mono specific to the Mono runtime
Projects
None yet
Milestone
Future
Development

No branches or pull requests
8 participants
@directhex @akoeplinger @stephentoub @karelz @maryamariyan @danmoseley @safern @MihaZupan