[.NET6 undocumented breaking change] HttpConnectionPool is throwing exception instead of returning HttpResponseMessage

[marekott]

Description

In .NET 6 there is a significant change in HttpConnectionPool. Exception will be thrown in palce where in .NET 5 HttpResponseMessage would be returned (https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs#L1553). It was introduced due to #48679. It is a major breaking change for my application because in previous versions after sending a request with HttpClient via proxy if we received response different than 200 we had conditional logic checking response to determine how to handle this situation. We were for example checking not only response status code but also headers that are now unaviable. Is there a workaround to preserve backward compatibility? Also was this change well documented? I believe that I didn't stumble upon it reading release notes and as pointed above it can be breaking change for some application.

Reproduction Steps

1. Register HttpClient using Proxy

services
    .AddHttpClient("HttpClientName")
    .ConfigurePrimaryHttpMessageHandler(sp =>
    {
        var handler = new HttpClientHandler
        {
            Proxy = new WebProxy(proxyAddress),
            UseProxy = true,
        };

        return handler;
    });

2. Using IHttpClientFactory resolve client and send request to desirable URL.

var response = await client.SendAsync(httpRequest, cancellationToken);

3. If proxy is for example acting as egress proxy and will not allow for that communication returning 403 (from proxy) exception will be thrown. Because of that all relevant for us informations are lost (except for response status code).

Expected behavior

HttpResponseMessage is returned intead of exception being thrown. It allows for processing that response.

Actual behavior

HttpRequestException is thrown and no HttpResponseMessage is avaible for processing.

Regression?

Expected behaviour was present in .NET 5.

Known Workarounds

Currently I am unaware of any workaround.

Configuration

No response

Other information

No response

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

In .NET 6 there is a significant change in HttpConnectionPool. Exception will be thrown in palce where in .NET 5 HttpResponseMessage would be returned (https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs#L1553). It was introduced due to #48679. It is a major breaking change for my application because in previous versions after sending a request with HttpClient via proxy if we received response different than 200 we had conditional logic checking response to determine how to handle this situation. We were for example checking not only response status code but also headers that are now unaviable. Is there a workaround to preserve backward compatibility? Also was this change well documented? I believe that I didn't stumble upon it reading release notes and as pointed above it can be breaking change for some application.

Reproduction Steps

1. Register HttpClient using Proxy

services
    .AddHttpClient("HttpClientName")
    .ConfigurePrimaryHttpMessageHandler(sp =>
    {
        var handler = new HttpClientHandler
        {
            Proxy = new WebProxy(proxyAddress),
            UseProxy = true,
        };

        return handler;
    });

2. Using IHttpClientFactory resolve client and send request to desirable URL.

var response = await client.SendAsync(httpRequest, cancellationToken);

3. If proxy is for example acting as egress proxy and will not allow for that communication returning 403 (from proxy) exception will be thrown. Because of that all relevant for us informations are lost (except for response status code).

Expected behavior

HttpResponseMessage is returned intead of exception being thrown. It allows for processing that response.

Actual behavior

HttpRequestException is thrown and no HttpResponseMessage is avaible for processing.

Regression?

Expected behaviour was present in .NET 5.

Known Workarounds

Currently I am unaware of any workaround.

Configuration

No response

Other information

No response

Author:	marekott
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[marekott]

@geoffkizer, @wfurt could you take a look?

[geoffkizer]

The rationale for this change is described in the linked issue, #48679. Given the rationale there, it's unlikely we would revert this.

We could certainly look at how to expose more information about the response received from the proxy here, or possibly provide some sort of callback to allow you to inspect the response yourself.

It would be helpful to have more information about what you are trying to do here. It sounds like you are trying to access various proxies and determine if they are usable or not? What responses do you expect and what response headers do you care about? What action, if any, do you take after a failure?

Possible workarounds:
(1) Use an http URI instead of https. The change only affects https URIs.
(2) Try to manually CONNECT to the proxy yourself. That is, create a request with URI=proxy URI and method=CONNECT, with Host header set to the host you want to connect to.

[ghost]

This issue has been marked needs-author-action since it may be missing important information. Please refer to our contribution guidelines for tips on how to report issues effectively.

[marekott]

@geoffkizer thank you for your answer. Providing an asynchronous callback would be a perfect soultion for my team.

In our scenario we are communicating with outside world via proxy which can based on some business rules cut off that communication by returning 403 to caller. If such thing happend we were examinig response headers and body to decide how to handle such situation. Now the only thing we have is exception with msg informing about proxy address and response status code. Choosing logic to perform based on exception message seems like an antipattern.

(1) Reaching some endpoints is only avaiable via https so it is not a solution for us
(2) You are providing us with framework for HttpClient communication via proxy on which we are heavy dependant and we would like to avoid such big change as switching to proposed solution. Like said before, callback for response will be sufficient.

PS I belive there is a typo in exception msg you are throwing in HttpConnectionPool (last character: ") . Would you like me to open seperate issue for that or make a PR fixing it?

runtime/src/libraries/System.Net.Http/src/Resources/Strings.resx

Line 592 in daf77fc

<value>The proxy tunnel request to proxy '{0}' failed with status code '{1}'."</value>

[alixiashu]

@geoffkizer Should we consider adding a setting toggle here to let dev select the pattern? Or could we add the status code to the HttpRequestException?
In our scenario, we do a test call for our service, then if get a response with 407, we could know that the user has a proxy and need an auth credential then we could tell the user to enter the credential to authorize the next operations. Could I just get the status code by a reg expression from the exception message based on

runtime/src/libraries/System.Net.Http/src/Resources/Strings.resx

Line 592 in daf77fc

<value>The proxy tunnel request to proxy '{0}' failed with status code '{1}'."</value>

The point here is that we don't know whether the user has a proxy or not, so your advice (2) is not applicable.

Besides, I am curious why we only do this update on HTTPS requests, but not HTTP requests？

[MihaZupan]

Besides, I am curious why we only do this update on HTTPS requests, but not HTTP requests？

See the detailed description here #48679 (comment)

With http, you intentionally opt out of security, and there's by-design no way to tell where the response actually came from.
With https, the proxy response to the CONNECT request is in cleartext, where as the actual response over the tunnel goes over TLS. A user making a request with HttpClient may reasonably expect that the response they observe came from the server over an authenticated tunnel, but before 6.0, it could have been generated by the proxy instead.

[wfurt]

Triage: it seems like we need to add documentation. This should be possibly solved by #66329.