Redirect conflict with uBlock Origin

[terrorist96]

Type: other

This extension failed to redirect a network request to https://s7.addthis.com/js/300/addthis_widget.js#pubid=tenthamendment because another extension (uBlock Origin) redirected it to chrome-extension://cjpalhdlnbpafiamejdnhcphjbkeiagm/web_accessible_resources/af3c919ee9a3d97f5dedaf2f8bec146a.javascript?secret=qqesz4qhqw5h.

Looks like a surrogate redirect conflict?

@gorhill

[blackhelicoptersdotnet]

I'm getting the same, in my case with a Google Analytics URL.

Appeared today with version 2018.2.26

This extension failed to redirect a network request to https://www.google-analytics.com/analytics.js because another extension (uBlock Origin) redirected it to chrome-extension://cjpalhdlnbpafiamejdnhcphjbkeiagm/web_accessible_resources/59fb24a2d12455d15bea20980e8a6801.javascript?secret=jblm6nqcej3g.

[ghost]

This extension failed to redirect a network request to https://s7.addthis.com/js/300/addthis_widget.js#pubid=ra-59c545b43f9cd35b because another extension (uBlock Origin) redirected it to chrome-extension://cjpalhdlnbpafiamejdnhcphjbkeiagm/web_accessible_resources/af3c919ee9a3d97f5dedaf2f8bec146a.javascript?secret=nrw0ygwf8axk.

[gorhill]

gorhill/uBlock#1170 (comment)

[jeremyn]

@gorhill Your comment says "The resources used by uBO for redirection purpose are found in this file." but the linked file, https://github.com/gorhill/uBlock/blob/master/assets/ublock/resources.txt , doesn't exist anymore. Is there an equivalent?

[gorhill]

I corrected the link, the resource files have been moved to a different repo since I wrote this comment: https://github.com/uBlockOrigin/uAssets/blob/master/filters/resources.txt.

[jeremyn]

Thanks @gorhill .

As an enhancement for HTTPS Everywhere, we might take resources.txt and use it to create exceptions for the various rewrites it does. For example we could add a checkbox that says "Exclude uBlock Origin rewrites" which, if checked, will not rewrite the items in resources.txt, and instead let uBlock Origin handle them.

I'm not saying this is a good idea to do, but it's something that could be done.

[mystica555]

I would appreciate this 'resources.txt' as exeptions, because honestly I don't want https everywhere to be redirecting me to tracking sites I explicitly block with ublock; the way it works now, trackers totally being blocked from loading, is the preferred scenario. I just want the ! to disappear making me think there are 'actual' problems with https on the site I am visiting.

[jeremyn]

@terrorist96 @blackhelicopters @ariksc @mystica555 Can you all please edit your comments to provide a URL where you go to see the problem? Not the error from the developer console, but the URL in your browser's address bar.

[terrorist96]

I'll try to post it if I come across it again. BTW, here's the PR Privacy Badger made to avoid this issue from occurring:
EFForg/privacybadger#1905

[gorhill]

@jeremyn

we might take resources.txt and use it to create exceptions for the various rewrites it does

In that case resources.txt is not the right place for this, that would rather be the filters themselves, those with the redirect= option in any of the filter lists in https://github.com/uBlockOrigin/uAssets/tree/master/filters. There is a lots of these, and these lists are constantly edited. For the specific case here, this would be this filter:

||addthis.com/*/addthis_widget.js$script,redirect=addthis.com/addthis_widget.js

Which in plain language means: redirect all network requests for addthis_widget.js from addthis.com to a local, neutered version of the script.

[madusmacus]

This extension failed to redirect a network request to https://www.google-analytics.com/ga.js because another extension (uBlock Origin)

I have also had

This extension failed to redirect a network request to https://s7.addthis.com/js/300/addthis_widget.js#pubid=tenthamendment because another extension (uBlock Origin) redirected it to chrome-extension://cjpalhdlnbpafiamejdnhcphjbkeiagm/web_accessible_resources

[jeremyn]

@gorhill Thanks. So it sounds like the logic for us might be: go through the files in https://github.com/uBlockOrigin/uAssets/tree/master/filters , look for filter lines that start with || and include redirect=, and add those partial-URLs to a list controlled by the checkbox I suggested earlier at #14961 (comment).

Why is this issue popular suddenly? Did something change recently in uBlock Origin?

[gorhill]

Why is this issue popular suddenly? Did something change recently in uBlock Origin?

I wondered. uBO has supported redirection since quite a while now, to data: URIs. However in a recent release uBO will redirect to web accessible resources instead of data: URIs. This has changed to solve long standing issue gorhill/uBlock#2823. Not sure whether this is what is causing more reports about this -- redirecting to data: URIs was also causing such conflicts before (as pointed out at old issue gorhill/uBlock#1170).

[jeremyn]

All of the reports here so far are reporting redirects to something under web_accessible_resources, so the recent uBlock Origin change may be related. But it sounds like these reporters should have seen the same error before, except it would have been to data: instead of web_accessible_resources, right?

[gorhill]

it would have been to data: instead of web_accessible_resources, right?

Yes, it's what I find puzzling.

[Jed-Tech]

Been getting this on two different computers this week. Both with this message:

This extension failed to redirect a network request to https://widgets.outbrain.com/outbrain.js because another extension (uBlock Origin) redirected it to chrome-extension:[LONGSTRING]/web_accessible_resources

[jeremyn]

Pinging @Bisaloo @brainwane @gloomy-ghost @Hainish @J0WI @wonderchook to let them know about this.

[HvyMtlChaos]

I'm also getting this error in my extensions, the variant that cites google analytics and uBlockOrigin. I don't want to unblock that url in UBO and tried to find a way to exclude it from HTTPSEverywhere and couldn't.

[Bisaloo]

Since this is only a warning and does not affect the functionality of either uBlock or HTTPS Everywhere, I was going to vote for wontfix. But after having a look at the comments on Chrome store, it looks like it's making many users worried.

I'm still not sure how to fix this though...

[languitar]

There's a very obvious indication by chrome that something is wrong, which I have never seen before. People have to be worried with that icon appearing.

[Bisaloo]

But it sounds like these reporters should have seen the same error before, except it would have been to data: instead of web_accessible_resources, right?

I did some digging on the Chromium bug tracker and it may have something to do with the following lines:

https://codereview.chromium.org/9857009/diff/1003/chrome/browser/extensions/api/web_request/web_request_api_helpers.cc

[cschanaj]

Possibly related https://bugs.chromium.org/p/chromium/issues/detail?id=800773

[gorhill]

does not affect the functionality of either uBlock or HTTPS Everywhere

It could affect uBO: if the request is redirect by HTTP-Everywhere first, this will prevent uBO from redirecting to a local neutered resource, meaning a network request meant to be blocked would no longer be blocked. On the other hand, if uBO redirects first, this won't affect the purpose of HTTPS-Everywhere.

[cucumbur]

It affects the functionality when you don't use a myopic technical lens. HTTPSeverywhere and uBlock origin are fighting - reddit nerds condescend about using some matrix plugin and various other hardcore solutions that aren't practical for average users. They stop using either one and leave bad reviews - likely on HTTPSEverywhere, which is where I had to go see the big red error after chrome made it seem like my extensions were exploding. Less people WILL use the extension as a result.
And if a programmer using HTTPS everywhere and ublock origin with decentraleyes and privacy badger disabled (I have different extension profiles), even I'm not likely to just move to privacy badger and leave ublock behind after customizing it and getting familiar with it, It's just the truth.

At the end of the day, I can make sure I'm using HTTPS when it matters most (and in general), I can't block my own ads (dev tools doesnt count) so I would drop HTTPS everywhere.

[gorhill]

(Copied from gorhill/uBlock#3639 (comment))

I won't tell people what to do. I will tell what is happening, and from this people can decide for themselves. One of two cases can happen:

The error message is a result of two extensions trying to redirect the same network request: the Chrome/WebExtensions API allows only one extension to redirect a network request. Extensions are not made aware that their redirection failed. The reason for this is probably technical: conceptually extensions execute in parallel.

uBO redirect wins

Benign.

Typically the error message "This extension failed to redirect a network request..." will be with HTTPS-Everywhere in the Extensions page (as reported in opening comment).

This won't affect the purpose of HTTPS-Everywhere -- because no network requests is fired to the outside world.

HTTPS-Everywhere redirect wins

Serious Benign.

Typically the error message "This extension failed to redirect a network request..." will be with uBlock Origin in the Extensions page.

This will prevent uBO from redirecting the network request to a local neutered resource, meaning a network request which was meant meant to be blocked by uBO would no longer be blocked. uBO won't be able to redirect the original http: network request, because this one will be redirected by HTTPS-Everywhere to a secure https: connection. However, uBO will properly redirect the https: request to its local neutered resource, because of course HTTPS-Everywhere won't touch this one.

Considerations

The order in which the extensions were installed is what decide which extension will redirect first (I can't remember the order the last one installed wins -- thanks @Bisaloo) -- so this means by re-installing one or the other you could move from the "serious" to the "benign" case. So mainly either ways all work as expected except that you will have to tolerate the warnings in the Extensions page.

When a secure page (https) tries to pull a script from an unsecure location (http), the browser will refuse, mixed active content has been forbidden by browsers since quite a while now. So in that case, HTTPS-Everywhere is not really needed.

Why is this error more widely reported now? I have no clue, uBO has been redirecting to neutered resources since a very long time now (see #1170). Somebody on HTTPS-Everywhere's issue tracker pointed out that something related to extension error was fixed in Chrome 65.

[cschanaj]

AFIAK, using both uBO and HTTPSE together has been causing issues for a long time due to the limitation in the webRequest APIs (namely only one extension is allowed to rewrite/ block a request, without notifying the extension losing the race). This is becoming an critical issue when Chrome 65 start displaying the runtime errors in the UI.

cc @Hainish as this might discourage the users from using HTTPSE...

@gorhill fixed. thank.

[gorhill]

only one extension is allowed to rewrite/ block a request

Blocking is not an issue -- if one extension orders a network request to be blocked, it will be blocked regardless of what other extensions decide. The issue is only when two or more extensions redirect. This is not an issue when the two extensions have similar purpose (uBO/Ghostery), but here we have two extensions with a different purpose.

[jeremyn]

Could someone please provide a URL where I can go to see the problem happen? I asked for this in #14961 (comment), but so far all the reports here just describe the error and not where they are seeing it.

[HvyMtlChaos]

bartovan, the same thing happened to mine. I'm guessing some update to one or both of the extensions caused this change.

uBlock Origin
This extension failed to redirect a network request to chrome-extension://cjpalhdlnbpafiamejdnhcphjbkeiagm/w

[Mactastic1-5]

I have been having this issue for weeks. I keep seeing it after I finish browsing.

[jeremyn]

Everyone: there is no need at this point to add a new comment simply to say you are getting the problem or to report the now-familiar This extension failed to redirect a network request to chrome-extension://.... Please instead thumbs-up the original comment if you are also getting the problem or want to show support for this issue.

[Jimbazuka]

I just want to point out the exact opposite happens to me where HTTPS everywhere "succeeds" in redirecting browser traffic over uBlockOrigin - but creates a clash that Chrome consider a failure/error: what I see is documented in here

[jeremyn]

To add to my previous comment: no one needs to add a comment to report an error happening in either direction, that is, This extension failed to redirect a network request to https://... or This extension failed to redirect a network request to chrome-extension://.... Please see @gorhill 's comment above at #14961 (comment) for a description of the problem.

I don't want to discourage people from showing their support for this issue, but rather, there is no need at the moment for anyone else to spend their time providing detailed information

[gorhill]

Consider starring this Chromium issue if you want a fix, I added a comment at the end:
https://bugs.chromium.org/p/chromium/issues/detail?id=111700#c31

[cowlicks]

I think it is worth noting the uBO change that is causing this, is actually a workaround for this chrome bug. Which has implications for other projects (like PB). So please also chime in there to get this fixed.

[Hainish]

@gorhill as a temporary resolution to this problem, can you let HTTPS Everywhere know that uBlock Origin is installed via onMessageExternal, and I will selectively disable the conflicting rulesets. We would have to determine in advance which URLs are potentially problematic, but this is known, correct? Meaning, it isn't dynamic, it's a static set of widget URLs which are locally proxied?

[jeremyn]

@Hainish See #14961 (comment) for a comment from @gorhill about getting a list of URLs.

[gorhill]

@Hainish This is not as simple as just providing a list of URLs. The redirection is decided based on two things:

• Whether the network request is meant to be blocked
• Whether the network request matches a pattern in one of the directives in the redirection database

For the first one, this is decided at webRequest.onBeforeRequest listener time. The exact same network request may be blocked most of the time, but then allowed on a handful of sites (those sites broken by the act of blocking). It's decided in real-time, not in a declarative way.

The second point is that a redirection is decided according to a pattern in the URL, so a list of URLs does not work here. For example, a redirection directive with pattern */b/ss/*&aqe=, which is meant to redirect to a local pixel image.

[Hainish]

@gorhill do you have statistical data on how often each of these patterns are matched in uBlock Origin? I'm wondering if we can avoid conflicts in a comprehensive but not complete way. We may be able to avoid the vast majority of conflict events in this way.

[jeremyn]

@Hainish If @gorhill doesn't have statistical data, then at least based on the comments here and gorhill/uBlock#3639 , specific handling for Google Analytics would take care of a lot of cases.

[Paradoxum]

Is there any way to stop the (!) icon appearing at the very least for us end users? I'm getting this constantly and it's so frustrating.

[gorhill]

I added a partial revert of the fix to gorhill/uBlock#2823 (which is what the redirection to "web accessible resources" was fixing). uBO will avoid to redirect to a "web accessible resource" and instead redirect to a data: URI when the two following conditions are present:

• uBO is running in a Chromium-based browser
• The URL to redirect is a non-secure http: one (i.e. one which HTTP-Everywhere will likely want to redirect)

[lightheaded]

Just wanted to leave a note here that I'm still occasionally receiving this error on HTTPS Everywhere 2018.4.11

[Hainish]

@lightheaded thanks, I've pinged Chromium again on this issue: https://bugs.chromium.org/p/chromium/issues/detail?id=111700#c37

[gorhill]

@lightheaded Having the exact web page where this happens would be useful to determine whether this is really the same exact issue identified here.

[lightheaded]

@gorhill it's hard to track as it seemingly happens at random. But today I caught it blocking something on Amazon:

This extension failed to redirect a network request to https://g-ecx.images-amazon.com/images/G/02/kindle/merch/2017/ACC/Promo/ACCBundles//UK_Bundle_Muscat_160X97.jpg because another extension (HTTPS Everywhere) redirected it to https://d1ge0kk1l5kms0.cloudfront.net/images/G/02/kindle/merch/2017/ACC/Promo/ACCBundles//UK_Bundle_Muscat_160X97.jpg

[gorhill]

@lightheaded I am not aware of any filters in uBO which would want to redirect the URL your are mentioning.

Also, I find it surprising that HTTPS Everywhere would want to redirect a https: URL to another https: URL. The one related rule in HTTPS-E is to redirect from http: to https::

• https-everywhere/src/chrome/content/rules/Images-Amazon.com.xml

  Lines 31 to 32 in 73c41f6

  	<rule from="^http://g-ecx\.images-amazon\.com/"
  	to="https://d1ge0kk1l5kms0.cloudfront.net/" />

In any case, assuming you created custom rules to cause the behavior you are observing on your side, as said above, uBO fix with regard to HTTPS-E is a mitigation, i.e. this should take care of most conflicts, but this won't take care of conflict arising as a result of HTTPS-E redirecting a https: URL to another https URL (which I believe shouldn't be happening normally?)

[jeremyn]

@gorhill FYI, there's an open issue #10843 to remove https:->https: redirects where possible.

[cschanaj]

It seems that chrome will stop notifying the users for extension conflicts in the dev channel.

This CL introduces the onActionIgnored event on the web request API which is
fired when an extension's proposed modification to a network request is ignored.
This happens in case of conflicts with other extensions. This should allow the
extensions to address conflicts as they like.

Also, we no longer notify the user in case of conflicts between extensions.

Ref

https://groups.google.com/a/chromium.org/forum/#!msg/extensions-reviews/YYn1etdxjgw/55bBFxxfCgAJ

https://developer.chrome.com/extensions/webRequest#event-onActionIgnored

[Hainish]

Fantastic! Thanks for the heads up @cschanaj - we should definitely handle this event.

[terrorist96]

Can this be closed? Marked as fixed since Aug 24.

Consider starring this Chromium issue if you want a fix, I added a comment at the end:
https://bugs.chromium.org/p/chromium/issues/detail?id=111700#c31

I haven't had this issue for a while.

[Libbum]

Yep, nothing on my systems for a while too.