Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

403 Forbidden #3

Open
qqqqqvb4 opened this issue Jun 27, 2024 · 23 comments
Open

403 Forbidden #3

qqqqqvb4 opened this issue Jun 27, 2024 · 23 comments

Comments

@qqqqqvb4
Copy link

It seems to me that over the past month Youtube has changed something. Both this and my own downloader no longer work. An attempt to download any livestream returns 403. After adding some missing parameters to the query we are able to get the correct status code but then we get this error: cabr.send_sabr_erro�. Any ideas?

@qqqqqvb4 qqqqqvb4 changed the title 403 Unauthorized 403 Forbidden Jun 27, 2024
@qqqqqvb4
Copy link
Author

Initial playback URL sent by browser:

https://rr2---sn-4ox-ixal.googlevideo.com/videoplayback?expire=1719534500&ei=Q699ZpKDPKTPi9oPk4Cv2Aw&ip=185.77.218.13&id=YpVL9Zz3kiY.1&itag=278&aitags=242%2C243%2C244%2C247%2C248%2C271%2C278&source=yt_live_broadcast&requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&mh=Fy&mm=44%2C29&mn=sn-4ox-ixal%2Csn-ixh7rn76&ms=lva%2Crdu&mv=m&mvi=2&pl=23&initcwndbps=1881250&siu=1&bui=AbKP-1NpHTjwCoYjgN017zNh5XDZ14-1gVKw5_CzJPV0Es9DVV-9_cKblgxkQULMpFTvwaUepg&spc=UWF9f8k5obvCG14-izXl2P5QrmXA7NMeTIiXply3UXjp2IIBRjI9BkcO0HLNb_tA-R3oKHNb9w&vprv=1&live=1&hang=1&noclen=1&svpuc=1&mime=video%2Fwebm&ns=Tq-aO7VADgT0kA_oxXnzIBQQ&rqh=1&gir=yes&mt=1719512432&fvip=2&keepalive=yes&c=WEB&sefc=1&n=49xlvCytdtPFxg&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cxpc%2Csiu%2Cbui%2Cspc%2Cvprv%2Clive%2Chang%2Cnoclen%2Csvpuc%2Cmime%2Cns%2Crqh%2Cgir&sig=AJfQdSswRgIhAJR881MxTAzOXE3IEtnWFHAd5gL4jiF70Y7fyCX4EnRvAiEApK5vS27a6rp7xUfY6oWZln7SXY5VDLycBRCzgdlZz6Y%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AHlkHjAwRAIgIK_XQDsZbdZd7YEJKGv8WSsVO5zgKkNlE5WBuq25BtYCIAS0ldnGBP7Svc2ZvpoAKhVq3VGNFX9NBnjr7SS13J19&alr=yes&cpn=GX_7nvwHW6kq7B29&cver=2.20240624.06.00&headm=1&rn=1&rbuf=0&pot=Ih_mPuY_gENJeNcO1g3fC9IM1gzXBtcH0QbTCN4M00Ka&ump=1&srfvp=1

Raw initial playback URL extracted from HTML:

https://rr2---sn-4ox-ixal.googlevideo.com/videoplayback?expire=1719536428&ei=zLZ9Zt7UO4iXv_IPqYGomAo&ip=185.77.218.13&id=YpVL9Zz3kiY.1&itag=278&aitags=242%2C243%2C244%2C247%2C248%2C271%2C278&source=yt_live_broadcast&requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&mh=Fy&mm=44%2C29&mn=sn-4ox-ixal%2Csn-ixh7rn76&ms=lva%2Crdu&mv=m&mvi=2&pl=23&initcwndbps=663750&bui=AbKP-1PvbaxUvi4_x_JPnUtClEZ5H3geAkNZsq07bZzj-AtHHCXM-uBh_NNJ5ck13eHWOYsFG8ZCoArB&spc=UWF9fx5x7mWz30J4BPBFc-jkh91CBaL50XgsSEbYj4_7vZy5zUyNoBBoPgtD&vprv=1&live=1&hang=1&noclen=1&svpuc=1&mime=video%2Fwebm&ns=915Rsavxg8VZSxjwIqQCguoQ&rqh=1&gir=yes&mt=1719514352&fvip=2&keepalive=yes&c=WEB&sefc=1&n=4Cw_l-oLmzOYOwRo&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cxpc%2Cbui%2Cspc%2Cvprv%2Clive%2Chang%2Cnoclen%2Csvpuc%2Cmime%2Cns%2Crqh%2Cgir&sig=AJfQdSswRQIhANs_ghOqLzgePvD60MLQjeQ_ZuxO5x-Ml8wXztLFdtUFAiBmLT7h5azsKpIekrMiqDXysh9CpNUE67LUkAgBFwRDMA%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AHlkHjAwRQIgN0nv-ydBQGp9XZzctLyl8Tw_lz99-CweDsnNjNqTn0ECIQCy4Y0IZwGYbievGEGV6f1_QMPCoIvIkHgOUnAvpSxcow%3D%3D

Differences after we add rn, rbuf, alr, cver, headm and cpn

EXPIRE
browser: 1719535839
soft: 1719536428

EI
browser: f7R9ZoSACYO56dsPp7mcuAo
soft: zLZ9Zt7UO4iXv_IPqYGomAo

INITCWNDBPS
browser: 1307500
soft: 663750

SIU
browser: 1
soft: null

BUI
browser: AbKP-1NXZ6Ht6itJgc0lw5wQafQxGtnOxBRtWyAtJGv57b-CHY7h2TNCWPrmuEsdgynvlvLd5Q
soft: AbKP-1PvbaxUvi4_x_JPnUtClEZ5H3geAkNZsq07bZzj-AtHHCXM-uBh_NNJ5ck13eHWOYsFG8ZCoArB

SPC
browser: UWF9f7icRx_zg8G3v7bUQc47AivH-y-R1M_Ff7-4jq6FA4DT41lyZVY6Qw3VFCJqSRdBkDRPyg
soft: UWF9fx5x7mWz30J4BPBFc-jkh91CBaL50XgsSEbYj4_7vZy5zUyNoBBoPgtD

NS
browser: 9N5UkGbJZ_sq3ufSadQMLtAQ
soft: 915Rsavxg8VZSxjwIqQCguoQ

MT
browser: 1719513865
soft: 1719514352

N
browser: sW80GDneUBknIA
soft: 4Cw_l-oLmzOYOwRo

SPARAMS
browser: expire,ei,ip,id,aitags,source,requiressl,xpc,siu,bui,spc,vprv,live,hang,noclen,svpuc,mime,ns,rqh,gir
soft: expire,ei,ip,id,aitags,source,requiressl,xpc,bui,spc,vprv,live,hang,noclen,svpuc,mime,ns,rqh,gir

SIG
browser: AJfQdSswRAIgeaaLzi3qfTzx5k7fgVJdzVWE5GHgEeNlEzyd-qLmJI0CICQV5Gy3rxb5tWJLSG7tZ7CNFGeqgwvrxnMke-KEKq_L
soft: AJfQdSswRQIhANs_ghOqLzgePvD60MLQjeQ_ZuxO5x-Ml8wXztLFdtUFAiBmLT7h5azsKpIekrMiqDXysh9CpNUE67LUkAgBFwRDMA==

LSIG
browser: AHlkHjAwRQIgVJemMZOeDcBeK20cfrWvBPlqVKElOkWlv21FoJJ_5B8CIQCrtVBca9rbYgDeLnKiCJ86M9uglNFHSAQ9YzVtlg9D5g==
soft: AHlkHjAwRQIgN0nv-ydBQGp9XZzctLyl8Tw_lz99-CweDsnNjNqTn0ECIQCy4Y0IZwGYbievGEGV6f1_QMPCoIvIkHgOUnAvpSxcow==

POT
browser: Ih-gcqBzxg8U85FCkEGZR5RAkECRSpFLl0qVRJhAlQ7c
soft: null

UMP
browser: 1
soft: null

SRFVP
browser: 1
soft: null

After adding the ump, we get the correct status code but we also get an error in the body cabr.send_sabr_erro�.

@qqqqqvb4
Copy link
Author

We can't change any of the parameters that are listed in the sparams because then we would also have to change the signature in the sig parameter.

@qqqqqvb4
Copy link
Author

The initial playback sent by browser has the same signature as the one that arrives in HTML so we are sure that the client in no way modifies and can not establish a new signature. So we can't modify the siu parameter for sure. The pot parameter (poToken) has existed for a long time and previously everything worked correctly without this parameter so I doubt that it is needed. So we can only modify ump and srfvp. In general, it's hard to find any resources because if someone is already downloading livestreams, they just use extra links to HLS or DASH instead of doing it the "native" way like a browser does

@qqqqqvb4
Copy link
Author

I also tested it with logged-in account sessions and on some accounts it just works.

@qqqqqvb4
Copy link
Author

Working initial playback extracted from HTML on logged-in session where everything works

https://rr1---sn-4ox-ixal.googlevideo.com/videoplayback?aitags=140&alr=yes&bui=AbKP-1MT-k-jmRKVteBr5XDcovq7e_azfbdNEi2aP6Nv0_zXFHjgxbtn-5_kY8bYIgIkWLQt0g&c=WEB&cpn=rWJZBorRNgieOudi&ctier=A&cver=2.20240626.07.00&ei=Snx-ZrX9CpbDv_IPsbOskAc&expire=1719586986&fvip=4&gir=yes&hang=1&headm=2&hightc=yes&id=bbRxU5rDzQg.1&initcwndbps=458750&ip=185.77.218.13&itag=140&keepalive=yes&live=1&lsig=AHlkHjAwRAIgFZuVjdRDD-wjFHXUD7DNhX1a-38jWXGIpyginwekvCECIG6ERlxs6UE1JSYj_-QjAQFIt02sJyMkzhmhtPK-TjGi&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&mh=nY&mime=audio%2Fmp4&mm=44%2C29&mn=sn-4ox-ixal%2Csn-ixh7yn7d&ms=lva%2Crdu&mt=1719564739&mv=m&mvi=1&n=gNUjJHNh4bQsSiIE&noclen=1&ns=HLH-FRJGCyqHoQCtD1Ah_08Q&pl=23&rbuf=0&requiressl=yes&rn=2&rqh=1&sefc=1&sig=AJfQdSswRgIhANUihI5wJI6VrSYhbHoeh_bhEuftPjVx1QDFNXqEM1siAiEAm-GvHW8bvAT5Lj91cBhz3yTVukJJQRidS41L3bOVEgQ%3D&siu=1&source=yt_live_broadcast&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cxpc%2Cctier%2Chightc%2Csiu%2Cbui%2Cspc%2Cvprv%2Clive%2Chang%2Cnoclen%2Csvpuc%2Cmime%2Cns%2Crqh%2Cgir&spc=UWF9f6W2Y4mIe9mMcAxIXPEf0Xih5Z2N0QRAfy3umMZqN5ZnBxVJyPx3CYkBr6WfWOAB7R5qHg&svpuc=1&vprv=1&xpc=EgVo2aDSNQ%3D%3D

working_yt

@qqqqqvb4
Copy link
Author

I already tested both formats with webm and mp4. Lottery what link will be returned for what session. Sometimes it works. Normal videos works always. The problem is only with livestreams. I will try to debug on the same html in the code as in the browser and find the cause. Weird as fuck

@qqqqqvb4
Copy link
Author

The difference in requests between software and browser with the same hardcoded HTML:

N
browser: N0IJaQE4bL6EXQ
soft: lJzm4zVoIdej2Upt

CPN
browser: 473xTk-1Mn1GX3EP
soft: 6wSIEhS3ykiaH2W1

POT
browser: Ih9nxWfEAbvk8Fb1V_Ze8FP3V_dW_Vb8UP1S81_3Urkb
soft: null

Still doesn't work

@qqqqqvb4
Copy link
Author

qqqqqvb4 commented Jun 28, 2024

I removed pot param in burp and everything still works. (but it gets a weird fallback from sequence to range?) cpn is simply a random string generated by the client and n comes ready in HTML from youtube so idk

@qqqqqvb4
Copy link
Author

@qqqqqvb4
Copy link
Author

qqqqqvb4 commented Jun 28, 2024

"As this issue seems to be coming from the context of YouTube ReVanced, I'd like to remind that the main purpose of PoToken is to ban unofficial YouTube apps and their users. If you can't play videos in YouTube ReVanced, that means everything works as intended. YouTube ReVanced and ReVanced in general are not supported or endorsed by microG."

@qqqqqvb4
Copy link
Author

https://github.com/unixfox/refresh-botguard-token-youtube

i tried:

const payload = {
  "videoId": "bbRxU5rDzQg",
    "context": {
    "client": {
      "clientName": "WEB",
      "clientVersion": "2.20240626.07.00",
      "clientFormFactor": "UNKNOWN_FORM_FACTOR",
      "browserVersion": "115.0",
      "browserName": "Firefox",
      "acceptHeader": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"
    },
    "thirdParty": {
      "embedUrl": "https://www.youtube.com"
    }
  }
}

fetch("https://www.youtube.com/youtubei/v1/player?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8", {
  "headers": {
    "accept": "*/*",
    "accept-language": "en-US,en;q=0.9",
    "authorization": "SAPISIDHASH 1719571630_ed81653b5707c00ee518b17c882c8f1afa55b44a",
    "cache-control": "no-cache",
    "content-type": "application/json",
    "pragma": "no-cache",
    "priority": "u=1, i",
    "sec-ch-ua": "\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\"",
    "sec-ch-ua-arch": "\"x86\"",
    "sec-ch-ua-bitness": "\"64\"",
    "sec-ch-ua-form-factors": "\"Desktop\"",
    "sec-ch-ua-full-version": "\"126.0.6478.114\"",
    "sec-ch-ua-full-version-list": "\"Not/A)Brand\";v=\"8.0.0.0\", \"Chromium\";v=\"126.0.6478.114\", \"Google Chrome\";v=\"126.0.6478.114\"",
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-model": "\"\"",
    "sec-ch-ua-platform": "\"Linux\"",
    "sec-ch-ua-platform-version": "\"6.1.0\"",
    "sec-ch-ua-wow64": "?0",
    "sec-fetch-dest": "empty",
    "sec-fetch-mode": "cors",
    "sec-fetch-site": "same-origin",
    "x-client-data": "CIi2yQEIo7bJAQipncoBCLvyygEIlqHLAQiHoM0BCMKFzgEI6ZPOAQjfm84BCLydzgEIxp3OAQiyns4BCLKfzgEI/qDOAQinos4BCNKizgEIj6XOAQjipc4BCOGnzgEY9MnNARjW680BGKCdzgE=",
    "x-goog-authuser": "0",
    "x-goog-visitor-id": "Cgt6YWpTSUVCSkdwbyiqqfqzBjIKCgJERRIEEgAgYA%3D%3D",
    "x-origin": "https://www.youtube.com",
    "x-youtube-ad-signals": "dt=1719571628162&flash=0&frm&u_tz=120&u_his=2&u_h=1080&u_w=1920&u_ah=1053&u_aw=1920&u_cd=24&bc=31&bih=924&biw=689&brdim=57%2C89%2C57%2C89%2C1920%2C27%2C1920%2C1055%2C689%2C924&vis=1&wgl=true&ca_type=image",
    "x-youtube-client-name": "1",
    "x-youtube-client-version": "2.20240626.07.00",
    "x-youtube-device": "cbr=Chrome&cbrver=126.0.0.0&ceng=WebKit&cengver=537.36&cos=X11&cplatform=DESKTOP",
    "x-youtube-identity-token": "QUFFLUhqa3FjbHVmeFJkdHpqZjV0UnFkWnoxLXhOd2haQXw=",
    "x-youtube-page-cl": "647127010",
    "x-youtube-page-label": "youtube.desktop.web_20240626_07_RC00",
    "x-youtube-time-zone": "Europe/Warsaw",
    "x-youtube-utc-offset": "120"
  },
  "referrer": "https://www.youtube.com/watch?v=bbRxU5rDzQg",
  "referrerPolicy": "origin-when-cross-origin",
  "body": JSON.stringify(payload),
  "method": "POST",
  "mode": "cors",
  "credentials": "include"
});

But there is no poToken in response.

@qqqqqvb4
Copy link
Author

yt-dlp/yt-dlp#10046 (comment)

"Actual YT web client adds a pot (proof of origin token / poToken) param to the videoplayback query, which is presumably sourced from Google's Botguard. Not much is documented about Botguard, but it seems to essentially be reCAPTCHA sans user input."

@qqqqqvb4
Copy link
Author

qqqqqvb4 commented Jun 28, 2024

I tried with debugger and manually generating a pot token and then pasting it to the software but the effect is still the same.

const visitorData = "CgtIYmxOUWdmUDM3SSiPovqzBjIiCgJTRRIcEhgSFhMLFBUWFwwYGRobHB0eHw4PIBAREiEgFA%3D%3D";
a.Bz.aX = visitorData;
a.Pz = visitorData;
let xd = (c = (b = lha(this, a)) != null ? b : mha(this, a)) != null ? c : nha(this, a);
console.log(xd);

base.js line ~8587

@qqqqqvb4
Copy link
Author

At worst, it looks like the pot token is being refreshed every few seconds/requests and is also required by the later videoplaybacks not only initial. If it actually uses a botguard underneath, this is the maximum and most annoying protection they could have introduced.

@qqqqqvb4
Copy link
Author

qqqqqvb4 commented Jun 28, 2024

I also tested in puppeter and there, too, after removing pot everything seems to work (Unlike the program. So I have no idea if the problem is actually poToken)

    if (pathBase === "videoplayback" || rawUrl.includes("videoplayback")) {
      const url = new URL(rawUrl);
      const searchParams = url.searchParams;
      searchParams.delete("pot");
      url.search = searchParams.toString();
      await request.continue({
        url: url.toString()
      })
      return;
    }

@qqqqqvb4
Copy link
Author

qqqqqvb4 commented Jun 28, 2024

It seems that player on /embed endpoint is much older. It works much more faster and smoothly, but there they also implemented poToken. It does not take videoplayback links directly from the HTML returned on /embed but sends an additional request to /player with generated token. I tried it this way:

https://www.youtube.com/embed/bbRxU5rDzQg

async function start() {
  const poToken = "Ili5Nbk030qTG_pSzQfYBYxd423sTO9YjF7jTNBHiGXAT_tf8Fz6UvNy6mfwVvxd3mb_XfR5_3fsYv9Czmz-Z9ZX8XeJUPFCjWXwd_hn_Fz8UvxknAb9EIpx";
  const videoId = "bbRxU5rDzQg";
  const visitorData = "CgtTSkZXQ1k1S0lvQSjb1PyzBjIiCgJGSRIcEhgSFhMLFBUWFwwYGRobHB0eHw4PIBAREiEgUw%3D%3D";

  const payload = {
    videoId: videoId,
    context: {
      client: {
        hl: "en",
        gl: "FI",
        clientName: "WEB",
        clientVersion: "2.20240628.01.00",
        visitorData: visitorData
      }
    },
    serviceIntegrityDimensions: {
      poToken: poToken
    }
  }
  const resp = await fetch("https://www.youtube.com/youtubei/v1/player?prettyPrint=false", {
    "headers": {
      "accept": "*/*",
      "accept-language": "en-US,en;q=0.9",
      "cache-control": "no-cache",
      "content-type": "application/json",
      "pragma": "no-cache",
      "priority": "u=1, i",
      "sec-ch-ua": "\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\"",
      "sec-ch-ua-arch": "\"x86\"",
      "sec-ch-ua-bitness": "\"64\"",
      "sec-ch-ua-form-factors": "\"Desktop\"",
      "sec-ch-ua-full-version": "\"126.0.6478.114\"",
      "sec-ch-ua-full-version-list": "\"Not/A)Brand\";v=\"8.0.0.0\", \"Chromium\";v=\"126.0.6478.114\", \"Google Chrome\";v=\"126.0.6478.114\"",
      "sec-ch-ua-mobile": "?0",
      "sec-ch-ua-model": "\"\"",
      "sec-ch-ua-platform": "\"Linux\"",
      "sec-ch-ua-platform-version": "\"6.1.0\"",
      "sec-ch-ua-wow64": "?0",
      "sec-fetch-dest": "empty",
      "sec-fetch-mode": "cors",
      "sec-fetch-site": "same-origin",
      "x-client-data": "CIi2yQEIo7bJAQipncoBCLvyygEIlKHLAQiHoM0BCMKFzgEI6ZPOAQjfm84BCLydzgEIxp3OAQiyns4BCLKfzgEI/qDOAQinos4BCNKizgEIj6XOAQjipc4BCOGnzgEY9MnNARjW680BGKCdzgE=",
      "x-goog-authuser": "0",
      "x-goog-visitor-id": "CgtIeDZuN0M5R3c0ayi8z_yzBjIKCgJERRIEEgAgQg%3D%3D",
      "x-origin": "https://www.youtube.com",
      "x-youtube-bootstrap-logged-in": "false",
      "x-youtube-client-name": "56",
      "x-youtube-client-version": "1.20240625.00.00",
      "Referer": "https://www.youtube.com/embed/bbRxU5rDzQg",
      "Referrer-Policy": "strict-origin-when-cross-origin"
    },
    body: JSON.stringify(payload),
    "method": "POST"
  });
  const json = await resp.json();
  const streamingData = json["streamingData"];
  const adaptiveFormats = streamingData["adaptiveFormats"];
  let worstVideo = null;
  for (const adaptiveFormat of adaptiveFormats) {
    if (worstVideo == null || adaptiveFormat.bitrate < worstVideo.bitrate) {
      worstVideo = adaptiveFormat;
    }
  }
  console.log("worst video", worstVideo);
  const url = worstVideo.url;
  const parsedUrl = new URL(url);
  const searchParams = parsedUrl.searchParams;
  searchParams.set("rbuf", "0")
  searchParams.set("headm", "2")
  searchParams.set("alr", "yes")
  searchParams.set("cver", "2.20240628.01.00")
  searchParams.set("cpn", "pt1gDFvRdxqJNtMs")
  searchParams.set("ump", "1")
  searchParams.set("srfvp", "1")
  searchParams.set("rn", "1")
  searchParams.set("pot", "Ili5Nbk030qTG_pSzQfYBYxd423sTO9YjF7jTNBHiGXAT_tf8Fz6UvNy6mfwVvxd3mb_XfR5_3fsYv9Czmz-Z9ZX8XeJUPFCjWXwd_hn_Fz8UvxknAb9EIpx")
  parsedUrl.search = searchParams.toString();
  const resp2 = await fetch(parsedUrl.toString(), {
    method: "POST",
    body: "x\u0000",
    headers: {
      "sec-ch-ua": "\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\"",
      "sec-ch-ua-arch": "\"x86\"",
      "sec-ch-ua-bitness": "\"64\"",
      "sec-ch-ua-form-factors": "\"Desktop\"",
      "sec-ch-ua-full-version": "\"126.0.6478.114\"",
      "sec-ch-ua-full-version-list": "\"Not/A)Brand\";v=\"8.0.0.0\", \"Chromium\";v=\"126.0.6478.114\", \"Google Chrome\";v=\"126.0.6478.114\"",
      "sec-ch-ua-mobile": "?0",
      "sec-ch-ua-model": "\"\"",
      "sec-ch-ua-platform": "\"Linux\"",
      "sec-ch-ua-platform-version": "\"6.1.0\"",
      "sec-ch-ua-wow64": "?0",
      "Referer": "https://www.youtube.com/",
      "Referrer-Policy": "origin-when-cross-origin"
    }
  })
  const body2 = await resp2.text();
  console.log(body2);
  console.log(resp2.headers);
  console.log(resp2.status)
}

start();

There is no difference. The effects are the same.

@qqqqqvb4
Copy link
Author

Okay, after numerous tests I am sure that it is not the fault of poToken. The pot is also almost certainly not generated by the Botguard because it takes too fast. poToken in the videoplayback and poToken from phone attestation are two different tokens. The problem is the parameter n. The browser changes the parameter and it is not the same as Youtube sends in HTML

@bashonly
Copy link

bashonly commented Jul 2, 2024

The n param needs to be decoded with a function found in youtube's player JS. Every player has a different decoding function. In the past, failure to decode the n sig would result in throttled transfer speeds (except for itags 17/18/22). As of a couple weeks ago, an incorrect n param now results in a 403 error. yt-dlp's code for doing this can be found here.

The pot param is something else. Not providing it won't result in an immediate error, but it's highly likely to be related to the recent IP/account block issues. My guess would be that Youtube tracks how many requests are made without a valid poToken, and, for the unlucky experiment group of the A/B test, making too many requests w/o pot results in IP/account blockage.

@qqqqqvb4
Copy link
Author

qqqqqvb4 commented Jul 2, 2024

After the first few requests pot gets longer so probably at the beginning some lighter version is fired up and after time when the page loads fully in the background the real Botguard is running and in fact it is rather as you say, the requests are validated after time. So yes, the fault was the lack of decoded (or not encoded - it's hard to say which way it works) parameter n.

If anyone is interested, my version looks like this:

var ytSigFixContentToRuntime = make(map[string]*goja.Runtime)
var ytSigFixContentToCallable = make(map[string]goja.Callable)
var ytSigFixMutex = new(sync.Mutex)
var fallbackYtSigFixContent, _ = os.ReadFile("./ytsigfix.js")

func decryptN(ytSigFixContent string, encryptedN string) (string, error) {
  ytSigFixMutex.Lock()
  defer ytSigFixMutex.Unlock()

  if ytSigFixContent == "" {
    ytSigFixContent = string(fallbackYtSigFixContent)
  }

  alreadyRuntime, ok := ytSigFixContentToRuntime[ytSigFixContent]
  if !ok {
    newRuntime := goja.New()
    _, runScriptErr := newRuntime.RunString(ytSigFixContent)
    if runScriptErr != nil {
      return "", runScriptErr
    }
    newEncryptFunction, exists := goja.AssertFunction(newRuntime.Get("encrypt"))
    if !exists {
      return "", fmt.Errorf("no encrypt function in script")
    }
    ytSigFixContentToRuntime[ytSigFixContent] = newRuntime
    ytSigFixContentToCallable[ytSigFixContent] = newEncryptFunction
    log.Printf("new script version: %s\n", ytSigFixContent)
  }

  alreadyEncryptFunction, _ := ytSigFixContentToCallable[ytSigFixContent]

  result, err := alreadyEncryptFunction(goja.Undefined(), alreadyRuntime.ToValue(encryptedN))
  if err != nil {
    return "", err
  }

  return result.String(), nil
}
func extractEncryptFunctionFromBaseJs(baseJs []byte) (string, error) {
  almostEndPhrase := []byte(`enhanced_except`)
  almostEndPhraseIndex := bytes.Index(baseJs, almostEndPhrase) + len(almostEndPhrase)
  if almostEndPhraseIndex < 0 {
    return "", fmt.Errorf("could not find enhanced except")
  }

  finalEndMaxIndex := almostEndPhraseIndex + 200

  cut := baseJs[almostEndPhraseIndex:finalEndMaxIndex]

  endPhrase := []byte(`return b.join("")`)
  endPhraseIndex := bytes.Index(cut, endPhrase) + len(endPhrase)
  if endPhraseIndex < 0 {
    return "", fmt.Errorf("could not find end phrase")
  }

  endIndex := almostEndPhraseIndex + endPhraseIndex

  cutFromEndToStartMax := baseJs[endIndex-5500 : endIndex]

  splitFunctionPhrase := []byte(`var b=a.split("")`)
  splitFunctionIndex := bytes.Index(cutFromEndToStartMax, splitFunctionPhrase)
  if splitFunctionIndex < 0 {
    return "", fmt.Errorf("could not find split function")
  }

  encryptStartIndex := endIndex - 5500 + splitFunctionIndex
  encryptEndIndex := endIndex

  encryptFunction := baseJs[encryptStartIndex:encryptEndIndex]

  ytSigFixContent := fmt.Sprintf("encrypt = function (a) {\n%s\n}", encryptFunction)

  return ytSigFixContent, nil
}

At first I totally forgot about this parameter and omitted it, because for somehow 2 years everything worked fine for me with this parameter omitted. And in my case, the difference with the correctly generated parameter and the one that youtube sends was not really important because my goal is to achieve view bots that download the worst possible quality. (Just the first few segments - youtube doesn't expect you to download them nonstop to sustain viewers)

@qqqqqvb4
Copy link
Author

qqqqqvb4 commented Jul 9, 2024

yt-dlp/yt-dlp#10390

there was another wave of IP bans / IPs being moved into the experiment group over the past few days

var playerJsUrlToExtractFunctionCache = make(map[string]string)
var playerJsUrlToExtractFunctionCacheLock = new(sync.Mutex)

func ExtractEncryptFunctionFromBaseJs(playerJsUrl string, baseJs []byte) (string, error) {
	playerJsUrlToExtractFunctionCacheLock.Lock()
	defer playerJsUrlToExtractFunctionCacheLock.Unlock()

	cachedExtractFunction, ok := playerJsUrlToExtractFunctionCache[playerJsUrl]
	if ok {
		return cachedExtractFunction, nil
	}

	almostEndPhrase := []byte(`enhanced_except`)
	almostEndPhraseIndex := bytes.Index(baseJs, almostEndPhrase)
	if almostEndPhraseIndex < 0 {
		return "", fmt.Errorf("could not find enhanced except")
	}
	almostEndPhraseIndex = almostEndPhraseIndex + len(almostEndPhrase)

	finalEndMaxIndex := almostEndPhraseIndex + 200

	cut := baseJs[almostEndPhraseIndex:finalEndMaxIndex]

	endPhrase := []byte(`return b.join("")`)
	endPhraseIndex := bytes.Index(cut, endPhrase)
	if endPhraseIndex < 0 {
		newEndPhrase := []byte(`return Array.prototype.join.call(b,"")`)
		newEndPhraseIndex := bytes.Index(cut, newEndPhrase)
		if newEndPhraseIndex >= 0 {
			log.Printf("new script version detected\n")
			newEndPhraseIndex = newEndPhraseIndex + len(newEndPhrase)
			endPhraseIndex = newEndPhraseIndex
		}
	} else {
		endPhraseIndex = endPhraseIndex + len(endPhrase)
	}
	if endPhraseIndex < 0 {
		return "", fmt.Errorf("could not find end phrase")
	}

	endIndex := almostEndPhraseIndex + endPhraseIndex

	cutFromEndToStartMax := baseJs[endIndex-5500 : endIndex]

	splitFunctionPhrase := []byte(`var b=a.split("")`)
	splitFunctionIndex := bytes.Index(cutFromEndToStartMax, splitFunctionPhrase)
	if splitFunctionIndex < 0 {
		newSplitFunctionPhrase := []byte(`var b=String.prototype.split.call(a,"")`)
		newSplitFunctionIndex := bytes.Index(cutFromEndToStartMax, newSplitFunctionPhrase)
		if newSplitFunctionIndex >= 0 {
			log.Printf("new script version detected\n")
			splitFunctionIndex = newSplitFunctionIndex
		}
	}
	if splitFunctionIndex < 0 {
		return "", fmt.Errorf("could not find split function")
	}

	encryptStartIndex := endIndex - 5500 + splitFunctionIndex
	encryptEndIndex := endIndex

	encryptFunction := baseJs[encryptStartIndex:encryptEndIndex]

	ytSigFixContent := fmt.Sprintf("encrypt = function (a) {\n%s\n}", encryptFunction)

	playerJsUrlToExtractFunctionCache[playerJsUrl] = ytSigFixContent

	log.Printf("new script version in cache for url: %s\n", playerJsUrl)

	return ytSigFixContent, nil
}

I don't know what these changes are due to. An attempt to fight scrapping bots that download videos to teach AI? or maybe just bumping up the version in some webpack or other vite like shit?

@7ERr0r
Copy link
Owner

7ERr0r commented Jul 9, 2024

Looks like there exists a working version of yt-dlp

yt-dlp/yt-dlp@04e17ba

@VORAPIS
Copy link

VORAPIS commented Jul 10, 2024

hi qqqqqvb4 did you find what cause error 403 for random people afterall?

@siraben
Copy link

siraben commented Sep 17, 2024

Hi all, I'm interested in using this project but I get the following messages, let me know if there is a fix.

[VID] download_format_segment  1: start GET
[AUD] download_format_segment  1: Status: 403 Forbidden (247ms since GET)
[AUD] Forbidden msg: "" // restart the program maybe?
[AUD] download_format_segment  1: start GET

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants