Change consent to pending to avoid youtube tracking #2207
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
unixfox
changed the title
SamantazFox
reviewed
Jun 25, 2021
| @@ -10,7 +10,7 @@ def add_yt_headers(request) | |||
| request.headers["x-youtube-client-name"] ||= "1" | |||
| request.headers["x-youtube-client-version"] ||= "2.20200609" | |||
| # Preserve original cookies and add new YT consent cookie for EU servers | |||
| request.headers["cookie"] = "#{request.headers["cookie"]?}; CONSENT=YES+" | |||
| request.headers["cookie"] = "#{request.headers["cookie"]?}; CONSENT=PENDING+#{Random.new.rand(900)}" | |||
There was a problem hiding this comment.
Watch out, the random number should be >=100 and <= 999!
See what the peeps at NewPipe did
There was a problem hiding this comment.
I don't understand how that happened but it can be over 1000, it is sometimes over 100 000
and so on… (you can search in NewPipeExtractor mocks)
There was a problem hiding this comment.
I don't understand how that happened but it can be over 1000, it is sometimes over 100 000
Oh, okay. I've never seen it being over 3 digits on youtube
There was a problem hiding this comment.
We should go with Random.new.rand(100..999) for now, so we can merge this PR and change that later, if needed...
|
Can you please rebase this changes on |
|
Youtube doesn't block the real HTML page anymore, so we probably may remove the CONSENT cookie at all: |
TheFrenchGhosty
added
the
unfinished
|
Closable? |
|
Like I said on matrix if you really want to close it then feel free to do it. It doesn't hurt to let it open. |
return42
pushed a commit
to return42/searxng
that referenced
this pull request
Jul 25, 2022
This change the previous bypass method for Google consent using ``ucbcb=1`` (6face21) to accept the consent using ``CONSENT=YES+``. The youtube_noapi and google have a similar API, at least for the consent[1]. Get CONSENT cookie from google reguest:: curl -i "https://www.google.com/search?q=time&tbm=isch" \ -A "Mozilla/5.0 (X11; Linux i686; rv:102.0) Gecko/20100101 Firefox/102.0" \ | grep -i consent ... location: https://consent.google.com/m?continue=https://www.google.com/search?q%3Dtime%26tbm%3Disch&gl=DE&m=0&pc=irp&uxe=eomtm&hl=en-US&src=1 set-cookie: CONSENT=PENDING+936; expires=Wed, 24-Jul-2024 11:26:20 GMT; path=/; domain=.google.com; Secure ... PENDING & YES [2]: Google change the way for consent about YouTube cookies agreement in EU countries. Instead of showing a popup in the website, YouTube redirects the user to a new webpage at consent.youtube.com domain ... Fix for this is to put a cookie CONSENT with YES+ value for every YouTube request [1] iv-org/invidious#2207 [2] TeamNewPipe/NewPipeExtractor#592 Closes: searxng#1432
kvch
pushed a commit
to kvch/searx
that referenced
this pull request
Jul 30, 2022
This change the previous bypass method for Google consent using ``ucbcb=1`` (6face21) to accept the consent using ``CONSENT=YES+``. The youtube_noapi and google have a similar API, at least for the consent[1]. Get CONSENT cookie from google reguest:: curl -i "https://www.google.com/search?q=time&tbm=isch" \ -A "Mozilla/5.0 (X11; Linux i686; rv:102.0) Gecko/20100101 Firefox/102.0" \ | grep -i consent ... location: https://consent.google.com/m?continue=https://www.google.com/search?q%3Dtime%26tbm%3Disch&gl=DE&m=0&pc=irp&uxe=eomtm&hl=en-US&src=1 set-cookie: CONSENT=PENDING+936; expires=Wed, 24-Jul-2024 11:26:20 GMT; path=/; domain=.google.com; Secure ... PENDING & YES [2]: Google change the way for consent about YouTube cookies agreement in EU countries. Instead of showing a popup in the website, YouTube redirects the user to a new webpage at consent.youtube.com domain ... Fix for this is to put a cookie CONSENT with YES+ value for every YouTube request [1] iv-org/invidious#2207 [2] TeamNewPipe/NewPipeExtractor#592 Closes: searxng/searxng#1432
kvch
added a commit
to searx/searx
that referenced
this pull request
Jul 30, 2022
* [fix] google engine: results XPath * [fix] google & youtube - set EU consent cookie This change the previous bypass method for Google consent using ``ucbcb=1`` (6face21) to accept the consent using ``CONSENT=YES+``. The youtube_noapi and google have a similar API, at least for the consent[1]. Get CONSENT cookie from google reguest:: curl -i "https://www.google.com/search?q=time&tbm=isch" \ -A "Mozilla/5.0 (X11; Linux i686; rv:102.0) Gecko/20100101 Firefox/102.0" \ | grep -i consent ... location: https://consent.google.com/m?continue=https://www.google.com/search?q%3Dtime%26tbm%3Disch&gl=DE&m=0&pc=irp&uxe=eomtm&hl=en-US&src=1 set-cookie: CONSENT=PENDING+936; expires=Wed, 24-Jul-2024 11:26:20 GMT; path=/; domain=.google.com; Secure ... PENDING & YES [2]: Google change the way for consent about YouTube cookies agreement in EU countries. Instead of showing a popup in the website, YouTube redirects the user to a new webpage at consent.youtube.com domain ... Fix for this is to put a cookie CONSENT with YES+ value for every YouTube request [1] iv-org/invidious#2207 [2] TeamNewPipe/NewPipeExtractor#592 Closes: searxng/searxng#1432 * [fix] sjp engine - convert enginename to a latin1 compliance name The engine name is not only a *name* its also a identifier that is used in logs, HTTP headers and more. Unicode characters in the name of an engine could cause various issues. Closes: searxng/searxng#1544 Signed-off-by: Markus Heiser <[email protected]> * [fix] engine tineye: handle 422 response of not supported img format Closes: searxng/searxng#1449 Signed-off-by: Markus Heiser <[email protected]> * bypass google consent with ucbcb=1 * [mod] Adds Lingva translate engine Add the lingva engine (which grabs data from google translate). Results from Lingva are added to the infobox results. * openstreetmap engine: return the localized named. For example: display "Tokyo" instead of "東京都" when the language is English. * [fix] engines/openstreetmap.py typo: user_langage --> user_language Signed-off-by: Markus Heiser <[email protected]> * Wikidata engine: ignore dummy entities * Wikidata engine: minor change of the SPARQL request The engine can be slow especially when the query won't return any answer. See https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual/MWAPI#Find_articles_in_Wikipedia_speaking_about_cheese_and_see_which_Wikibase_items_they_correspond_to Co-authored-by: Léon Tiekötter <[email protected]> Co-authored-by: Emilien Devos <[email protected]> Co-authored-by: Markus Heiser <[email protected]> Co-authored-by: Emilien Devos <[email protected]> Co-authored-by: ta <[email protected]> Co-authored-by: Alexandre Flament <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a PR that is part of fixing #2191.
We don't have to accept the YouTube consent, we can set it to pending with a random number after, and it works.
The major drawback of doing that is that we need to append
ucbcb=1to every request that requires the consent to be either pending (PENDING) or accept (YES).I tried to cover all the needed URLs, but I probably missed some of them. So that's why the PR is still a draft and I need help for finding all the part of the code that I would have missed.