Sql Connection fails to connect after unscheduled SQL Server shutdown.

[buzzaII]

Describe the bug

Have a sample application and when running in Docker on the latest .Net Core 3.1 image (3.1.8 at the time of this writing) and using the BackgroundService pattern in .net core, when connecting to SQLConnection in a continuous while loop with a very small delay, there seems to be times where the background service just throws exceptions about connection pool when trying to reconnect after the SQL has an unscheduled shutdown (reboot without graceful shutdown).

When running on Windows (not docker) this works as expected.

Sample Application: TestSqlConnectionOpen.zip

System.InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at Microsoft.Data.SqlClient.SqlConnection.Open()
   at TestSqlConnectionOpen.Worker2.ExecuteAsync(CancellationToken stoppingToken) in C:\xxxxx\xxxxxxxx\xxxxxxx\xxxxxxx\TestSqlConnectionOpen\TestSqlConnectionOpen\Worker2.cs:line 34
Exception thrown: 'System.InvalidOperationException' in Microsoft.Data.SqlClient.dll
�[41m�[30mfail�[39m�[22m�[49m: TestSqlConnectionOpen.Worker3[0]
      Error Worker Worker3 with Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
System.InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at Microsoft.Data.SqlClient.SqlConnection.Open()

To reproduce

While running the attached solution with VS and Docker integration, you will see that if you reboot the SQL server with the following command shutdown -f -r -t 0 the application doesn't recover when the server comes back up.

In Windows the application recovers almost immediately.

Expected behavior

Docker + Linux: have the sample reconnect to SQL server as it does in Windows, almost immediately

Further technical details

Microsoft.Data.SqlClient version: 1.1.2
.NET target: Core 3.1.2
SQL Server version: Microsoft SQL Server 2017 (RTM-CU19) (KB4535007) - 14.0.3281.6 (X64) Jan 23 2020 21:00:04 Copyright (C) 2017 Microsoft Corporation Developer Edition (64-bit) on Windows Server 2016 Standard 10.0 (Build 14393: ) (Hypervisor)
Operating system: Docker container - see attached DockerFile

[cheenamalhotra]

Hi @buzzaII

We have similar issue #359 reported regarding Connection Resiliency in Unix/MacOS, it seems related to the same issue.