Support absolute-form and authority-form URLs by web server

[asvetlov]

Fixes #6227

[asvetlov]

@webknjaz I think many proxy tests were literally wrong.
reqest.path_qs is for path with query_string and fragment parts, not for absolute URL

[codecov]

Codecov Report

Merging #6409 (f74a39b) into master (15941aa) will increase coverage by 0.00%.
The diff coverage is 97.77%.

@@           Coverage Diff           @@
##           master    #6409   +/-   ##
=======================================
  Coverage   93.34%   93.35%           
=======================================
  Files         104      104           
  Lines       30456    30485   +29     
  Branches     3064     3067    +3     
=======================================
+ Hits        28429    28458   +29     
  Misses       1850     1850           
  Partials      177      177           

Flag	Coverage Δ
unit	93.27% <97.77%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
tests/test_proxy_functional.py	62.68% <90.90%> (-0.08%)	⬇️
aiohttp/http_parser.py	92.73% <100.00%> (+0.07%)	⬆️
aiohttp/web_request.py	96.00% <100.00%> (+0.05%)	⬆️
tests/test_http_parser.py	99.07% <100.00%> (+0.01%)	⬆️
tests/test_web_request.py	99.09% <100.00%> (+0.01%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 15941aa...f74a39b. Read the comment docs.

[webknjaz]

@webknjaz I think many proxy tests were literally wrong.

Yeah, I noticed some oddities but haven't looked into it closely besides making what was failing pass. One of the things I've filed an issue for, as a result of that, was #6031 — have you seen it? I think it's something that you could have better context about than me.
FWIW I think it'd be more reliable to have the tests rewritten as end-to-end using the proxy.py fixture that is now implemented in the tests rather than hoping that the mocks are correct.

[asvetlov]

Rewriting tests with proxy.py is a great idea. Sorry, I have no resources for performing it myself.
Regarding mocking in affected tests: test_utils.py is not touched. I mean the real request should have very similar behavior to created by make_mocked_request() factory.

[patchback]

Backport to 3.8: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply c3356ac on top of patchback/backports/3.8/c3356ac8acd5bc656a2894fc2eab260a5a2f2087/pr-6409

Backporting merged PR #6409 into master

1. Ensure you have a local repo clone of your fork. Unless you cloned it
   from the upstream, this would be your origin remote.
2. Make sure you have an upstream repo added as a remote too. In these
   instructions you'll refer to it by the name upstream. If you don't
   have it, here's how you can add it:

   $ git remote add upstream https://github.com/aio-libs/aiohttp.git

3. Ensure you have the latest copy of upstream and prepare a branch
   that will hold the backported code:

   $ git fetch upstream
   $ git checkout -b patchback/backports/3.8/c3356ac8acd5bc656a2894fc2eab260a5a2f2087/pr-6409 upstream/3.8

4. Now, cherry-pick PR Support absolute-form and authority-form URLs by web server #6409 contents into that branch:

   $ git cherry-pick -x c3356ac8acd5bc656a2894fc2eab260a5a2f2087

   If it'll yell at you with something like fatal: Commit c3356ac8acd5bc656a2894fc2eab260a5a2f2087 is a merge but no -m option was given., add -m 1 as follows intead:

   $ git cherry-pick -m1 -x c3356ac8acd5bc656a2894fc2eab260a5a2f2087

5. At this point, you'll probably encounter some merge conflicts. You must
   resolve them in to preserve the patch from PR Support absolute-form and authority-form URLs by web server #6409 as close to the
   original as possible.
6. Push this branch to your fork on GitHub:

   $ git push origin patchback/backports/3.8/c3356ac8acd5bc656a2894fc2eab260a5a2f2087/pr-6409

7. Create a PR, ensure that the CI is green. If it's not — update it so that
   the tests and any other checks pass. This is it!
   Now relax and wait for the maintainers to process your pull request
   when they have some cycles to do reviews. Don't worry — they'll tell you if
   any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

[patchback]

Backport to 3.9: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply c3356ac on top of patchback/backports/3.9/c3356ac8acd5bc656a2894fc2eab260a5a2f2087/pr-6409

Backporting merged PR #6409 into master

1. Ensure you have a local repo clone of your fork. Unless you cloned it
   from the upstream, this would be your origin remote.
2. Make sure you have an upstream repo added as a remote too. In these
   instructions you'll refer to it by the name upstream. If you don't
   have it, here's how you can add it:

   $ git remote add upstream https://github.com/aio-libs/aiohttp.git

3. Ensure you have the latest copy of upstream and prepare a branch
   that will hold the backported code:

   $ git fetch upstream
   $ git checkout -b patchback/backports/3.9/c3356ac8acd5bc656a2894fc2eab260a5a2f2087/pr-6409 upstream/3.9

4. Now, cherry-pick PR Support absolute-form and authority-form URLs by web server #6409 contents into that branch:

   $ git cherry-pick -x c3356ac8acd5bc656a2894fc2eab260a5a2f2087

   If it'll yell at you with something like fatal: Commit c3356ac8acd5bc656a2894fc2eab260a5a2f2087 is a merge but no -m option was given., add -m 1 as follows intead:

   $ git cherry-pick -m1 -x c3356ac8acd5bc656a2894fc2eab260a5a2f2087

5. At this point, you'll probably encounter some merge conflicts. You must
   resolve them in to preserve the patch from PR Support absolute-form and authority-form URLs by web server #6409 as close to the
   original as possible.
6. Push this branch to your fork on GitHub:

   $ git push origin patchback/backports/3.9/c3356ac8acd5bc656a2894fc2eab260a5a2f2087/pr-6409

7. Create a PR, ensure that the CI is green. If it's not — update it so that
   the tests and any other checks pass. This is it!
   Now relax and wait for the maintainers to process your pull request
   when they have some cycles to do reviews. Don't worry — they'll tell you if
   any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

[asvetlov]

Backported by 17ba1f0 and 535c58e