feat(StreamsExtractor): generate PoToken

[FineFindus]

Implements support for locally generating PoTokens using the device web view. This is a direct port of
TeamNewPipe/NewPipe#11955 to native Kotlin.
However, there are some notable differences:

• NewPipe use an additional PoTokenGenerator interface. As there is only one implementation right now, this is skipped
• NewPipe handles broken web views on older devices. Since we're increasing minSdk in BREAKING CHANGE: bump minSdk to 26 #7047, I'm hoping we can get away without

It is marked as a draft, since while it does generate valid PoTokens, playback does not work (Source error: open failed: ENOENT; help is appreciated).

Closes: #7065

[FineFindus]

Please don't add unnecessary comments. Linking an issue to be closed is fine, but the rest just adds noise. I'm already well aware of the NewPipe PR (see description above).

[Figim]

Please don't add unnecessary comments. Linking an issue to be closed is fine, but the rest just adds noise. I'm already well aware of the NewPipe PR (see description above).

Sorry.

1. "Source Error" only occurs if "HLS" is enabled. DASH works great. tested

2. Live streams also play smoothly on HLS

[Bnyro]

The crash log is a temporary DNS issue on your side, that's not LibreTube related.

[Figim]

A source error still occurs when starting a video in HLS.

XRecorder_20250205_01.mp4

[FineFindus]

A few questions:

• Since HLS does not work with potokens either, should it be removed/disabled?
• JavaScriptUtil.kt uses nanojson, should it be ported as well? I left it for now, as it would make it easier to follow future changes. There isn't any noticable impact on filesize, since it is already included in NewPipeExtractor.

[YT-Advanced]

• Since HLS does not work with potokens either, should it be removed/disabled?

This issues seem to be NewPipeExtractor bug itself, since HLS don't need pot but it still be appended in streaming url

• JavaScriptUtil.kt uses nanojson, should it be ported as well? I left it for now, as it would make it easier to follow future changes.

Uhm u should port it also

[gechoto]

With the current code the purpose of JavaScriptUtil.kt is not perfectly clear at first look and it might take a while to notice that this is for po tokens.
The class and function names are pretty generic.
For example in the first function it says something about some "challenge data". Yeah cool, which challenge data? Someone new to the project won't know.

I suggest to do one of these to make it more obvious it is about po tokens:

• Move JavaScriptUtil.kt and PoTokenWebView.kt to a sub-package named potoken
• Rename JavaScriptUtil.kt to something like PoTokenJavaScriptUtil.kt
• Rename the functions and extend the documentation in JavaScriptUtil.kt to make it clear this is for po tokens

NewPipe did the first option here and put everything po token related into a sub package:
https://github.com/TeamNewPipe/NewPipe/tree/dev/app/src/main/java/org/schabi/newpipe/util/potoken

[Bnyro]

Since HLS does not work with potokens either, should it be removed/disabled?

It shouldn't be completely removed because it might still work with some private Piped instances, however disabling it by default is possible.

JavaScriptUtil.kt uses nanojson, should it be ported as well? I left it for now, as it would make it easier to follow future changes.

Yes, I would like to keep the used libraries consistent across the code. Since the NanoJSON code part is only very small and the code differs already due to the fact that it's ported to Kotlin, I think it's reasonable to port it.

I don't have much time to look into the problems you mentioned at the moment (and the next few weeks) unfortunately due to tons of exams at university. I'll see if I can find some time to allocate for reviewing soon, but no guarantees.

[FineFindus]

Disabled and ported.

I don't have much time to look into the problems you mentioned at the moment (and the next few weeks) unfortunately due to tons of exams at university. I'll see if I can find some time to allocate for reviewing soon, but no guarantees.

Sure, no pressure. I would also like to clean up the commits before merging. Good luck with your exams.

[gechoto]

use JsonNull instead of JsonPrimitive("") to make it consistent

the other values (like JsonPrimitive(messageId)) are also returned as JsonNull if they are null

from nanojson value function:

Emits a string value (or null) with a key.

so I guess JsonNull better matches the previous/NewPipe code

[gechoto]

There should be some better error handling.
Currently if something in parseChallengeData or parseIntegrityTokenData fails the app crashes.

[gechoto]

There is a NewPipe PR which ports po token code to coroutines

TeamNewPipe/NewPipe#12028

Maybe this could be used as a base.
With the code in the linked PR the app does not crash.
It does handle exceptions better.

[FineFindus]

Currently if something in parseChallengeData or parseIntegrityTokenData fails the app crashes.

Please attach a crash log/reproducible case and don't post vague comments about 'something' crashing.

[gechoto]

Currently if something in parseChallengeData or parseIntegrityTokenData fails the app crashes.

Please attach a crash log/reproducible case and don't post vague comments about 'something' crashing.

One specific crash log won't be useful. It crashes on any error.
Just do something obviously wrong in parseChallengeData, could be corrupted challenge data or for example try to access something which does not exist:

val messageId = challengeData[10].jsonPrimitive.content