.NET Core 3.0 Docker Container Won't Connect to SQL Server

[jeffgbradley2]

.NET Core 3.0 Docker Container Won't Connect to SQL Server

We have upgraded some of our .NET Core 2.2 applications to .NET Core 3.0. After the upgrade, our applications running in Docker containers fail to connect to SQL Server 2016 running on a standalone machine in our domain network. Reviewing the packet trace, we see that the connection hangs at the Pre-Login handshake, and eventually times out. Switching back to .NET Core 2.2 resolves the issue immediately, and our containers can again connect to our databases.

We have obviously verified that there are no DNS issues, since these containers are running in production with .NET Core 2.2. .NET Core 3.0 breaks them as they cannot connect to SQL Server over the network, so we are rolling back to 2.2 for now and terminating the upgrade until we can determine why .NET Core 2.2 containers connect to our SQL Servers but .NET Core 3.0 containers do not.

We have also created new .NET Core 3.0 projects from scratch to test the connection. The result is the same: the containers time out during Pre-Login to SQL Server. Changing the project back to .NET Core 2.2, along with EF Core back to 2.2.6, and the issue is immediately resolved.

As we are using EF Core for data access, we have also tried leaving EF Core at 2.2.6 but upgrading the project to .NET Core 3.0, but the connection still times out.

If we do not add Docker support and run the apps as Console or Web apps (not containerized), then there are no issues with .NET Core 3.0. It is only when we containerize them that .NET Core 3.0 apps fail to connect to SQL Server.

Per suggestion in Stack Overflow, I changed the data access in the test project to Microsoft.Data.SqlClient using hand-written SQL and a DataReader. I then recreated the Dockerfile for .net core 3.0 and verified that the correct container image was referenced. Same behavior: hung at PreLogin handshake with SQL Server. Switched to .net core 2.2 and was able to connect and read data from SQL Server.

[stijn-lcp]

We have the same issue:
Microsoft.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: TCP Provider, error: 35 - An internal exception was caught) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL. ---> Interop+Crypto+OpenSslCryptographicException: error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol

[scalablecory]

@MichaelSimons might have some insight.

[vcsjones]

@StijnStaelens What docker container image is being used?

[jeffgbradley2]

mcr.microsoft.com/dotnet/core/aspnet:3.0-buster-slim

and

mcr.microsoft.com/dotnet/core/runtime:3.0-buster-slim

both hang on pre-login handshake, switching back to their 2.2 counterparts is only solution we have been able to find.

here is the top section from Dockerfile from a Web API container that will not connect unless we flip it back to 2.2:

FROM mcr.microsoft.com/dotnet/core/aspnet:3.0-buster-slim AS base
WORKDIR /app
EXPOSE 80

FROM mcr.microsoft.com/dotnet/core/sdk:3.0-buster AS build

[vcsjones]

@bartonjs could this be Buster raising their OpenSSL TLS security level to 2 for OpenSSL 1.1?

[vcsjones]

So that image has:

[system_default_sect]
MinProtocol = TLSv1.2
CipherString = DEFAULT@SECLEVEL=2

in /etc/ssl/openssl.cnf.

I would be curious if adding this to your Dockerfile improves the situation:

RUN sed -i 's/DEFAULT@SECLEVEL=2/DEFAULT@SECLEVEL=1/g' /etc/ssl/openssl.cnf

Then rebuild your image and run.

This may explain the error that @StijnStaelens is seeing more than a hang.

[bartonjs]

Yep, it's probably the SECLEVEL.

[vcsjones]

Alternatively, you could try a different Docker image base like Ubuntu Bionic, which isn't quite as fast-paced as Debian is to more aggressive TLS configuration.

[stijn-lcp]

@vcsjones I tried the extra command, but it still doesn't work. My dockerfile now looks like this
`
FROM mcr.microsoft.com/dotnet/core/sdk:3.0 AS builder
RUN sed -i 's/DEFAULT@SECLEVEL=2/DEFAULT@SECLEVEL=1/g' /etc/ssl/openssl.cnf
WORKDIR src

COPY ./project.csproj .
RUN dotnet restore ./project.csproj
COPY . .
RUN dotnet publish ./project.csproj --configuration Release

FROM mcr.microsoft.com/dotnet/core/aspnet:3.0
RUN sed -i 's/DEFAULT@SECLEVEL=2/DEFAULT@SECLEVEL=1/g' /etc/ssl/openssl.cnf
COPY --from=builder src/bin/Release/netcoreapp3.0/publish/ app
WORKDIR app

ENTRYPOINT ["dotnet", "project.dll"]
`

My code works fine when I run it on windows, in Docker it fails

[stijn-lcp]

Using mcr.microsoft.com/dotnet/core/aspnet:3.0-bionic worked.

[vcsjones]

@StijnStaelens hm. You're using different base images from @jeffgbradley2, I'll examine those.

What version of SQL server are you connecting to?

[stijn-lcp]

@vcsjones,
We tried with 2 servers. 1 is 'Microsoft SQL Server Web 2017 (64-bit)'. The other is a 'Microsoft SQL Server Web 2012 (64-bit)'. Both failed. These are both self-hosted and we try to connect with this connectionstring:
Password=xxxxx;Persist Security Info=True;User ID=xxxxxx;Initial Catalog=xxxxxx;Data Source=1xx.xxx.xxx.xxx;

[vcsjones]

@StijnStaelens can you please try:

RUN sed -i 's/DEFAULT@SECLEVEL=2/DEFAULT@SECLEVEL=1/g' /etc/ssl/openssl.cnf
RUN sed -i 's/MinProtocol = TLSv1.2/MinProtocol = TLSv1/g' /etc/ssl/openssl.cnf
RUN sed -i 's/DEFAULT@SECLEVEL=2/DEFAULT@SECLEVEL=1/g' /usr/lib/ssl/openssl.cnf
RUN sed -i 's/MinProtocol = TLSv1.2/MinProtocol = TLSv1/g' /usr/lib/ssl/openssl.cnf

[jeffgbradley2]

We have confirmed that switching to Bionic fixes the issue.

[jeffgbradley2]

Switching to bionic fixed our issue for asp.net core, but not for console containers. For console containers, after switching to bionic we get the following:

It was not possible to find any compatible framework version
The specified framework 'Microsoft.AspNetCore.App', version '3.0.0' was not found.
  - No frameworks were found.

You can resolve the problem by installing the specified framework and/or SDK.

Here is snippet from Dockerfile:

FROM mcr.microsoft.com/dotnet/core/runtime:3.0-bionic AS base
WORKDIR /app

FROM mcr.microsoft.com/dotnet/core/sdk:3.0-bionic AS build
WORKDIR /src

[MichaelSimons]

@jeffgbradley2, can you make sure you have the latest version of the images pulled locally? If you have a previous version cached, you must explicitly pull the images to get any updates.

Another way to validate would be to run docker run -it --rm mcr.microsoft.com/dotnet/core/runtime:3.0-bionic dotnet --list-runtimes

[jeffgbradley2]

This is happening locally and in our CICD pipeline.

Running that command is what I and our Devops group expect to see.

[MichaelSimons]

@jeffgbradley2 - I'm confused. You said:

For console containers, after switching to bionic we get the following:

It was not possible to find any compatible framework version
The specified framework 'Microsoft.AspNetCore.App', version '3.0.0' was not found.

• No frameworks were found.

You can resolve the problem by installing the specified framework and/or SDK.

What do you mean by console container? Are you saying it is a containerized console app? Why is your console app depending on aspnet?

[MichaelSimons]

@jeffgbradley2 , I'm not following. Your screen shot shows Microsoft.NETCore.app which is what I expect for a console app. The error you included was for the Microsoft.AspNetCore.App.

[aidanjryan]

This looks to me like there are two different issues here. The original report is about a hang. A different person (@StijnStaelens ) reported a pre-login handshake error.

I think it's doubtful that TLS implementation would be responsible for the hang, although I guess it's possible. The hang is probably the same as #22, #165, or #201

We're experiencing this hang and it's stopping us from deploying .NET Core 3 across our infra.

[aidanjryan]

@jeffgbradley2 your CI troubleshooting sure is making a lot of noise in this issue, and an error about "not possible to find any compatible framework version" sure isn't related to the issue re: connecting to SQL server.

[stijn-lcp]

@StijnStaelens can you please try:

RUN sed -i 's/DEFAULT@SECLEVEL=2/DEFAULT@SECLEVEL=1/g' /etc/ssl/openssl.cnf
RUN sed -i 's/MinProtocol = TLSv1.2/MinProtocol = TLSv1/g' /etc/ssl/openssl.cnf
RUN sed -i 's/DEFAULT@SECLEVEL=2/DEFAULT@SECLEVEL=1/g' /usr/lib/ssl/openssl.cnf
RUN sed -i 's/MinProtocol = TLSv1.2/MinProtocol = TLSv1/g' /usr/lib/ssl/openssl.cnf

@vcsjones , I now used the standard image mcr.microsoft.com/dotnet/core/aspnet:3.0 (that didn't work before) and added those commands. Now it seems to work.

[MichaelSimons]

Should this issue have been closed or should it be left open? The end user experience doesn't feel optimal to me. This is going to affect a lot of users, how are they going to know how to address the problem? Downgrading the seclevel feels like a patch. What changes need to be made to work with seclevel2?

[vcsjones]

What changes need to be made to work with seclevel2?

This is ultimately a problem of the SQL server that is being connected to. The snippet I posted does two things: it lowers the required TLS version, and adjusts the SECLEVEL down to one.

For changing the minimum TLS version from 1.2 to 1.0, if this fixed the problem, then the SQL server needs to be patched and configured to use TLSv1.2 as described in this support article. I'm a bit out of my comfort zone with SQL Server + TLS.

If the SECLEVEL change was the source of the fix, then then this could be a few things, such as RC4 being used, or DHE-1024 getting used.

@StijnStaelens If you could help answer one more question, that'd be great. Given these two fixes:

1:

RUN sed -i 's/DEFAULT@SECLEVEL=2/DEFAULT@SECLEVEL=1/g' /etc/ssl/openssl.cnf
RUN sed -i 's/DEFAULT@SECLEVEL=2/DEFAULT@SECLEVEL=1/g' /usr/lib/ssl/openssl.cnf

and

RUN sed -i 's/MinProtocol = TLSv1.2/MinProtocol = TLSv1/g' /etc/ssl/openssl.cnf
RUN sed -i 's/MinProtocol = TLSv1.2/MinProtocol = TLSv1/g' /usr/lib/ssl/openssl.cnf

Do either of these resolve the issue without the other? That is, can you use 1 without 2 and it still works? Can you use 2 without 1 and does it work? Or does it only work with both 1 and 2?

[stijn-lcp]

@vcsjones

RUN sed -i 's/MinProtocol = TLSv1.2/MinProtocol = TLSv1/g' /etc/ssl/openssl.cnf
RUN sed -i 's/MinProtocol = TLSv1.2/MinProtocol = TLSv1/g' /usr/lib/ssl/openssl.cnf

These commands alone worked.

[aderderian]

I am seeing this issue with mcr.microsoft.com/dotnet/core/aspnet:3.1-nanoserver-1903 . I can't connect to a remote SQL server. In my case it is Azure. Any updates on this issue? I want to run a windows container.

[marcelbeeker]

My problem is solved. I have added an egress rule in my Calico config. Now I can connect to both the internet as SQL Server without any problem. Outlook voor Android downloaden<https://aka.ms/ghei36>

…

________________________________ From: Arra Derderian <[email protected]> Sent: Thursday, May 7, 2020 4:45:24 AM To: dotnet/SqlClient <[email protected]> Cc: marcelbeeker <[email protected]>; Mention <[email protected]> Subject: Re: [dotnet/SqlClient] .NET Core 3.0 Docker Container Won't Connect to SQL Server (#222) I am seeing this issue with mcr.microsoft.com/dotnet/core/aspnet:3.1-nanoserver-1903 . I can't connect to a remote SQL server. In my case it is Azure. Any updates on this issue? I want to run a windows container. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2FSqlClient%2Fissues%2F222%23issuecomment-624996925&data=02%7C01%7C%7Cca599e1188f4408eea3e08d7f230b193%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637244163254963048&sdata=ellUCuPEMyIjPcy%2B1dNcc9kSGwzsuS63F%2B5wRy9nu0A%3D&reserved=0>, or unsubscribe<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAENYSNSHPQO6SZ5WFVXWBX3RQIOEJANCNFSM4I26TWZQ&data=02%7C01%7C%7Cca599e1188f4408eea3e08d7f230b193%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637244163254963048&sdata=XZZHOMExVB2wRTTAJJ9jKaEHQ%2BP9J14D%2F2gD4HK4Kuo%3D&reserved=0>.

[aderderian]

My issue seems to be related to the GlobalProtect gateway as well. It only works when I am on my VPN. Somehow this adapter is conflicting with DNS even when I disable it or change the interface metric order.

[canertosuner]

Any update about it ?
I tried using 3.1-bionic image but I got the same exception when there is huge number of request (like 3000 per min.)

A connection was successfully established with the server, but then an error occurred during the pre-login handshake.

[NorDroN]

Hi!
I have also created new .NET Core 3.1 projects to test the connection to my DB (Microsoft SQL Server 2008 R2 SP1). The result: the containers time out during Pre-Login to SQL Server. I changed the project to .NET Core 2.2 with EF Core 2.2.6, but it didn't help me.

[hermanho]

It works without modification on
mcr.microsoft.com/dotnet/core/aspnet:3.1-bionic
but does not work on
mcr.microsoft.com/dotnet/core/aspnet:3.1-focal

[canertosuner]

It works without modification on

mcr.microsoft.com/dotnet/core/aspnet:3.1-bionic

but does not work on

mcr.microsoft.com/dotnet/core/aspnet:3.1-focal

Make a load test just like I did. 200-300 request per second and you will see that it will start throwing handshake exception. But if your app does not consume such number of request most probably you will not see any exception.

[aderderian]

I suggest people look to see if they have a VPN connection or another connection that could potentially be messing with this. I had a GlobalProtect VPN that must have had some routing rules in place even when disabled. I have to have it on for it to work now, despite changing the order of the connection. I didn't have time to solve it further, this was the fix for now.

[NorDroN]

It works correct without docker.
It doesn't work in mcr.microsoft.com/dotnet/core/aspnet:3.1-bionic too.

[hermanho]

It works without modification on
mcr.microsoft.com/dotnet/core/aspnet:3.1-bionic
but does not work on
mcr.microsoft.com/dotnet/core/aspnet:3.1-focal

Make a load test just like I did. 200-300 request per second and you will see that it will start throwing handshake exception. But if your app does not consume such number of request most probably you will not see any exception.

I tried with jmeter (number of threads is 300). It still works without exception. My app is a form submitting app which insert the form data to sql through ef core. It is running on aks with 1 pod and my sql server is "Microsoft SQL Server 2012 (SP3)".

I tried to re-build and deploy the image with 3.1-focal and it is stopped working in the first round.

[niennguyen84]

Use mcr.microsoft.com/dotnet/core/aspnet:3.1-bionic instead of
mcr.microsoft.com/dotnet/core/aspnet:3.1-buster
It works for me

[felp95]

use this image FROM mcr.microsoft.com/dotnet/core/runtime:3.1-bionic AS base
works for me too thx

[leili-ncut]

FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-bionic
It works for me !

[Leonardo-Ferreira]

Im having the same issue. Im using the mcr.microsoft.com/dotnet/core/aspnet:3.1 image and trying to connect to a 2012 SQL Server. Problem started after upgrading from .net core 2.1 to 3.1. On my local machine (windows) I cannot reproduce the issue!

[felp95]

It Works for me thanks

[Atilyus]

FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-bionic
It works for me too

[Saibamen]

For mcr.microsoft.com/dotnet/aspnet:5.0-buster-slim image I needed to add

RUN sed -i 's/DEFAULT@SECLEVEL=2/DEFAULT@SECLEVEL=1/g' /etc/ssl/openssl.cnf
RUN sed -i 's/MinProtocol = TLSv1.2/MinProtocol = TLSv1/g' /etc/ssl/openssl.cnf
RUN sed -i 's/DEFAULT@SECLEVEL=2/DEFAULT@SECLEVEL=1/g' /usr/lib/ssl/openssl.cnf
RUN sed -i 's/MinProtocol = TLSv1.2/MinProtocol = TLSv1/g' /usr/lib/ssl/openssl.cnf

EDIT:
If you have enabled TLS 1.2 on SQL Server, you need only to change CipherString (DEFAULT@SECLEVEL=2 to DEFAULT@SECLEVEL=1). IMO this is more secure than downgrading to TLS 1.0 protocol from 1999...

[cadaov]

We have confirmed that switching to Bionic fixes the issue.

Sadly so far there's no official image for Net 5.
But with this #222 (comment) it worked

[cheenamalhotra]

Sadly so far there's no official image for Net 5.

Below official images are available for .NET 5:

• mcr.microsoft.com/dotnet/sdk:5.0
• mcr.microsoft.com/dotnet/aspnet:5.0

ref: Official .NET Docker images

[Saibamen]

@cheenamalhotra There are no docker images with .NET 5 and Bionic...

[cheenamalhotra]

It is true .NET SDK 5+ are not supporting bionic anymore, but focal images have been available for a while now and with all necessary server and ssl certificate updates, clients should be able to connect via focal images [Troubleshooting Guide]. The most common issue everyone faced with .NET 5 is the update to default cipher suite, for which solutions are available.

Is there a problem that has not been solved yet for TLS 1.2?

Because, everyone would have to do this update sometime to continue upgrading .NET SDK, so not having a bionic image to work with that allows insecure TLS protocols is not going to be an option to block upgrade in future.

I would highly recommend solving environment issues NOW using .NET Core 3.1 focal images and unblocking your applications to be able to upgrade to newer .NET 5+ SDK.

If there are any issues that the SQL Server supports TLS 1.2 but focal images cannot connect and none of the above solutions can solve it, please open a new issue to get the right attention.

[thinkOfaNumber]

For those arriving late to the party, none of these options seemed to work for me (neither openssl.cnf nor the 3.1-buster image).

It turns out my problem was the default instance that was listening on 1433 was not the instance that I wanted. I was getting the pre-login handshake error, but with the weird error code 0 - Success! (This is not the error you're looking for)...
Microsoft.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: TCP Provider, error: 0 - Success)

I had to change the default SQL Server instance to use dynamic ports, restart the instance, then change the named instance I wanted to use port 1433, restart that instance, and then everything was great.

Except that I still couldn't use docker run -p 127.0.0.1:1433:1433 I had to use docker run -p 1433:1433 otherwise I got yet another error:
docker: Error response from daemon: Ports are not available: listen tcp 127.0.0.1:1433: bind: An attempt was made to access a socket in a way forbidden by its access permissions. time="2021-08-25T09:47:24+09:30" level=error msg="error waiting for container: context canceled"

Hope this helps someone following.

[rimvn]

Any update about it ? I tried using 3.1-bionic image but I got the same exception when there is huge number of request (like 3000 per min.)

A connection was successfully established with the server, but then an error occurred during the pre-login handshake.

I faced the same problem. How did you solve it?@canertosuner