User-Agent sniffing broken by Chrome 100 and Firefox 100

[ato]

Describe the bug

Pywb sniffs the user-agent in RewriterWithJSProxy.ua_allows_obj_proxy to decide whether it can use object proxies for JS rewriting. This appears to have broken with Chrome 100 and Firefox 100:

$ curl -s -H'User-Agent: Chrome/99' http://localhost:8080/test/20220411050413mp_/http://example.org/test.html | grep 123
foo.location = 123;
$ curl -s -H'User-Agent: Chrome/100' http://localhost:8080/test/20220411050413mp_/http://example.org/test.html | grep 123
foo.WB_wombat_location = 123;
$ curl -s -H'User-Agent: Firefox/99' http://localhost:8080/test/20220411050413mp_/http://example.org/test.html | grep 123
foo.location = 123;
$ curl -s -H'User-Agent: Firefox/100' http://localhost:8080/test/20220411050413mp_/http://example.org/test.html | grep 123
foo.WB_wombat_location = 123;

This causes many pages in the wild to redirect to a bogus URL on load including:

https://www.climate200.com.au/
https://www.news.com.au/
https://www.abc.net.au/news/

Steps to reproduce the bug

cat >test.html <<EOF
  <script>
    let foo = {};
    foo.location = 123;
  </script>
  <h1>OK</h1>
EOF

warcit -o http://example.org/ test.html
wb-manager init test
wb-manager add test test.html.warc.gz
pywb &
google-chrome-stable http://localhost:8080/test/http://example.org/test.html

Expected output: OK
Actual output: URL Not Found http://example.org/123

Environment

Working: Linux Chrome v99.0.4844.84
Not working: Linux Chrome v100.0.4896.60
Not working: Linux Chrome v100.0.4896.75

Working: Linux Firefox 99.0
Not working: Linux Firefox Developer Edition 100.0b2

Additional Context

The user agent parsing is done by werkzeug.useragents.UserAgent and seems to be deprecated and removed in newer versions werkzeug.

[ikreymer]

Thanks for reporting! Was considering keeping old version of werkzeug, but maybe that's not the best solution given this issue..

[ato]

werkzeug developers suggest using https://github.com/ua-parser/uap-python instead in pallets/werkzeug#2078

Was also thinking about just inverting the browser sniffing check so that object proxies are used by default and the non-proxy rewrites are only used on browsers known to not support them (MSIE, Chrome < 49, Firefox < 44 etc).

[ikreymer]

werkzeug developers suggest using https://github.com/ua-parser/uap-python instead in pallets/werkzeug#2078

Was also thinking about just inverting the browser sniffing check so that object proxies are used by default and the non-proxy rewrites are only used on browsers known to not support them (MSIE, Chrome < 49, Firefox < 44 etc).

Yep, was thinking the same thing, working on a fix.

[ikreymer]

Fixed in the 2.6.6 release!

[ato]

Retested our problem sites with pywb 2.6.6 and indeed fixed. Thanks very much. :-)