feat: Support redirects in WebDAV service

[dtretyakov]

Currently if WebDAV service uses redirects it's impossible to use sccache due to lack of redirects support in opendal (mozilla/sccache#1749).

In OpenDAL core redirects are disabled by default: https://github.com/apache/incubator-opendal/blob/40a0d04bd8081a52e61470450112d66bc8975b1c/core/src/raw/http_util/client.rs#L63-L64

In our case it's fine to support HTTP 302/307 redirects for GET requests.

[Xuanwo]

Should be handled inside webdav services it self.

[VladRassokhin]

Should be handled inside webdav services it self.

Per section 12 of WebDAV RFC:

Note also that WebDAV servers are
known to use 300-level redirect responses (and early interoperability
tests found clients unprepared to see those responses).

Anyway, inability to modify redirection policy from the outside is limiting.
Let's at least change code to not override redirect policy in HttpClient.build if it was specified (e.g. !=Policy::default()).

[Xuanwo]

Let's at least change code to not override redirect policy in HttpClient.build if it was specified (e.g. !=Policy::default()).

I mean redirect should be handled by serivce itself (aka opendal:services:Webdav). Service should check and redirect to location at needs.

[Xuanwo]

Actions

• Don't return error while meeting 302/307 in read, redirect to new location instead

[dtretyakov]

Please avoid typical security issue during redirect implementation: if "authority" (hostname + port) in redirect location differs from the value in original request, don't send original Authorization header value for redirect location.

[Gnosnay]

pls let me try on this. @Xuanwo

[Gnosnay]

Please avoid typical security issue during redirect implementation: if "authority" (hostname + port) in redirect location differs from the value in original request, don't send original Authorization header value for redirect location.

Hi @dtretyakov . May i have one example for redirecting HTTP response? What does location look like in this case?

[dtretyakov]

@Yansongsongsong it looks like that:

Request
URL: https://host:port/path/to/object?query=param
Method: GET
Authorization: xxx
...

Response 
Status Code: 307
Content-Length: 0
Location: https://cdn-host/path/to/object?query=param&Expires=123&Signature=xxx

And then subsequent request should use Location from the response to fetch the actual data.

[dtretyakov]

@Yansongsongsong do we have estimate when redirects will be supported?

I'm asking because it's critical to start using sccache in our setup (mozilla/sccache#1749)

[Gnosnay]

@Yansongsongsong do we have estimate when redirects will be supported?

I'm asking because it's critical to start using sccache in our setup (mozilla/sccache#1749)

@dtretyakov hi Tretyakov, sorry for the late. Because I had some private staff to handle, there was no progress in last week until this Monday.
I have finished the code now and begin to have one integration test.

It may be released in this week. thank you!

[Gnosnay]

Hi anyone can help add feat: as prefix of title of this issue?

The PR pipeline is failed due to wrong prefix name of this issue. but i can not modify the title of this issue.

Thank you so much!

[dtretyakov]

@Yansongsongsong renamed, please feel free to proceed.

[Xuanwo]

The PR pipeline is failed due to wrong prefix name of this issue. but i can not modify the title of this issue.

Sorry, it's related to our CI setup. I'm working on the fix.

[Xuanwo]

Fixed by #2256

[dtretyakov]

@Xuanwo am I right, that it's not a part of v0.36.0 release? If so, could we please include it the next bugfix release, e.g. 0.36.1?

[Xuanwo]

Yes, we will release this feature in next patch release.