Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Add fallback to TextDecoder and TextEncoder #4324

Merged
merged 3 commits into from
Jul 11, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions build/conformance.textproto
Original file line number Diff line number Diff line change
Expand Up @@ -333,3 +333,25 @@ requirement: {
"See also https://bit.ly/3wAsoj5"
whitelist_regexp: "node_modules/"
}

# Disallow the general use of TextDecoder and TextEncoder, which is not
# available on all supported platforms.
requirement: {
type: BANNED_NAME_CALL
value: "TextDecoder"
value: "window.TextDecoder"
error_message:
"Using \"TextDecoder\" directly is not allowed; "
"because is not supported on Xbox and old browsers."
whitelist_regexp: "lib/util/string_utils.js"
}

requirement: {
type: BANNED_NAME_CALL
value: "TextEncoder"
value: "window.TextEncoder"
error_message:
"Using \"TextEncoder\" directly is not allowed; "
"because is not supported on Xbox and old browsers."
whitelist_regexp: "lib/util/string_utils.js"
}
2 changes: 0 additions & 2 deletions demo/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,6 @@
<script defer src="../node_modules/tippy.js/umd/index.min.js"></script>
<!-- Improved PWA capability on mobile Safari is enabled by loading pwacompat: -->
<script defer src="../node_modules/pwacompat/pwacompat.min.js"></script>
<!-- TextDecoder polyfill, required for legacy Edge. -->
<script defer src="../node_modules/fastestsmallesttextencoderdecoder/EncoderDecoderTogether.min.js"></script>

<!-- Include IMA SDK to enable client-side ad insertion: -->
<script type="text/javascript" src="https://imasdk.googleapis.com/js/sdkloader/ima3.js"></script>
Expand Down
1 change: 1 addition & 0 deletions docs/tutorials/upgrade.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ application:
- TextDecoder/TextEncoder platform support or polyfill required (affects
Xbox One, but not evergreen browsers); we suggest the polyfill
[https://github.com/anonyco/FastestSmallestTextEncoderDecoder](fastestsmallesttextencoderdecoder/EncoderDecoderTogether.min.js)
Fallback included by default in v4.2

- Support removed:
- IE11 support removed
Expand Down
69 changes: 55 additions & 14 deletions lib/util/string_utils.js
Original file line number Diff line number Diff line change
Expand Up @@ -36,19 +36,38 @@ shaka.util.StringUtils = class {
if (uint8[0] == 0xef && uint8[1] == 0xbb && uint8[2] == 0xbf) {
uint8 = uint8.subarray(3);
}

// Use the TextDecoder interface to decode the text. This has the advantage
// compared to the previously-standard decodeUriComponent that it will
// continue parsing even if it finds an invalid UTF8 character, rather than
// stop and throw an error.
const utf8decoder = new TextDecoder();
const decoded = utf8decoder.decode(uint8);
if (decoded.includes('\uFFFD')) {
shaka.log.alwaysError('Decoded string contains an "unknown character" ' +
'codepoint. That probably means the UTF8 ' +
'encoding was incorrect!');
if (window.TextDecoder) {
// Use the TextDecoder interface to decode the text. This has the
// advantage compared to the previously-standard decodeUriComponent that
// it will continue parsing even if it finds an invalid UTF8 character,
// rather than stop and throw an error.
const utf8decoder = new TextDecoder();
const decoded = utf8decoder.decode(uint8);
if (decoded.includes('\uFFFD')) {
shaka.log.alwaysError('Decoded string contains an "unknown character' +
'" codepoint. That probably means the UTF8 ' +
'encoding was incorrect!');
}
return decoded;
} else {
// http://stackoverflow.com/a/13691499
const utf8 = shaka.util.StringUtils.fromCharCode(uint8);
// This converts each character in the string to an escape sequence. If
// the character is in the ASCII range, it is not converted; otherwise it
// is converted to a URI escape sequence.
// Example: '\x67\x35\xe3\x82\xac' -> 'g#%E3%82%AC'
const escaped = escape(utf8);
// Decode the escaped sequence. This will interpret UTF-8 sequences into
// the correct character.
// Example: 'g#%E3%82%AC' -> 'g#€'
try {
return decodeURIComponent(escaped);
} catch (e) {
throw new shaka.util.Error(
shaka.util.Error.Severity.CRITICAL, shaka.util.Error.Category.TEXT,
shaka.util.Error.Code.BAD_ENCODING);
}
}
return decoded;
}


Expand Down Expand Up @@ -141,8 +160,30 @@ shaka.util.StringUtils = class {
* @export
*/
static toUTF8(str) {
const utf8Encoder = new TextEncoder();
return shaka.util.BufferUtils.toArrayBuffer(utf8Encoder.encode(str));
if (window.TextEncoder) {
const utf8Encoder = new TextEncoder();
return shaka.util.BufferUtils.toArrayBuffer(utf8Encoder.encode(str));
} else {
// http://stackoverflow.com/a/13691499
// Converts the given string to a URI encoded string. If a character
// falls in the ASCII range, it is not converted; otherwise it will be
// converted to a series of URI escape sequences according to UTF-8.
// Example: 'g#€' -> 'g#%E3%82%AC'
const encoded = encodeURIComponent(str);
// Convert each escape sequence individually into a character. Each
// escape sequence is interpreted as a code-point, so if an escape
// sequence happens to be part of a multi-byte sequence, each byte will
// be converted to a single character.
// Example: 'g#%E3%82%AC' -> '\x67\x35\xe3\x82\xac'
const utf8 = unescape(encoded);

const result = new Uint8Array(utf8.length);
for (let i = 0; i < utf8.length; i++) {
const item = utf8[i];
result[i] = item.charCodeAt(0);
}
return shaka.util.BufferUtils.toArrayBuffer(result);
}
}


Expand Down
13 changes: 0 additions & 13 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 0 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@
"eslint-config-google": "^0.14.0",
"eslint-plugin-shaka-rules": "file:./build/eslint-plugin-shaka-rules",
"esprima": "^4.0.1",
"fastestsmallesttextencoderdecoder": "^1.0.22",
"fontfaceonload": "^1.0.2",
"google-closure-compiler-java": "^20220301.0.0",
"google-closure-deps": "https://gitpkg.now.sh/google/closure-library/closure-deps?d7736da6",
Expand Down Expand Up @@ -71,7 +70,6 @@
"dialog-polyfill/dist/dialog-polyfill.js",
"eme-encryption-scheme-polyfill/index.js",
"es6-promise-polyfill/promise.min.js",
"fastestsmallesttextencoderdecoder/EncoderDecoderTogether.min.js",
"google-closure-library/closure/goog/base.js",
"less/dist/less.js",
"material-design-lite/dist/material.indigo-blue.min.css",
Expand Down