ts.convertToBase64 does not handle emojis

[dsherret]

Bug Report

TypeScript/src/compiler/utilities.ts

Line 5244 in da00ba6

export function convertToBase64(input: string): string {

> ts.convertToBase64("📣❓")
'7aC97bOj4p2T'
> Buffer.from("📣❓").toString("base64")
'8J+To+Kdkw=='

🔎 Search Terms

• ts.convertToBase64
• emojis

🕗 Version & Regression Information

Probably a long time.

⏯ Playground Link

N/A

💻 Code

ts.convertToBase64("📣❓")

🙁 Actual behavior

7aC97bOj4p2T

🙂 Expected behavior

8J+To+Kdkw==

[RyanCavanaugh]

AFAIK this is only used to encode source map contents, which themselves cannot contain emoji. Switching these to a non-ordinal iteration is definitely going to make it slower -- without a concrete upside (or if I'm wrong about how this is called), I think this is a Won't Fix

[RyanCavanaugh]

Reading the inbound link, I'm guessing I'm wrong about whether sourcemaps can contain emoji

[bartlomieju]

We used implementation of base64 encoder from deno_std (which itself is based on https://gist.github.com/enepomnyaschih/72c423f727d395eeaa09697058238727) and it fixed the issue:
https://github.com/denoland/deno/blob/4e1ca1d1787f25ab9f0a72ccf9d8028f70b3a7ed/cli/tsc/99_main_compiler.js#L75-L110

[dsherret]

By the way, when running on Node it uses ts.sys.base64encode, which uses Buffer.from(...).toString("base64"). So only non-Node environments are affected (issue doesn't happen with tsc on Node).

[RyanCavanaugh]

Gotcha, we can make the fallback implementation as slow as needed

[jakebailey]

I'm looking for everyone who's patching typescript in prep for TS 5.0 (where that sort of thing breaks), and found this issue via deno's code.

@dsherret Can you instead provide base64encode on the host instead, thus avoiding the patch? Or are there calls which somehow don't use the host?

[bartlomieju]

@jakebailey we do provide base64encode on the host and we don't have a TypeScript patch for this functionality.

[jakebailey]

I'm confused; https://github.com/denoland/deno/blob/main/cli/tsc/99_main_compiler.js#L822 appears to be patching TypeScript at runtime (and links back here).

[jakebailey]

That being said, we now emit ES2018, so, maybe there's an efficient way to do this now anyhow.

[jakebailey]

Then again, there's also a ts.deno object, so maybe this isn't the actual ts API: https://github.com/denoland/deno/blob/4c6db7aa1493139f5a832c1e9ebfe44a1c80af80/cli/tsc/99_main_compiler.js#L323

[dsherret]

We don't use typescript for source maps anymore so we don't need that. I've been meaning to delete that code.

Then again, there's also a ts.deno object, so maybe this isn't the actual ts API: https://github.com/denoland/deno/blob/4c6db7aa1493139f5a832c1e9ebfe44a1c80af80/cli/tsc/99_main_compiler.js#L323

I've also been meaning to discuss about this, but I've been too busy. (For the future: Essentially we have a fork of TS to make npm specifiers work—we have multiple globalThis objects... one for Deno and one for Node. The ts.deno is to make the fork easier to maintain)

[jakebailey]

Ah, cool; just wanted to make sure that you weren't runtime patching and got surprised when 5.0 caused that to throw (since ESM exports can't be modified externally, bundler or not).

Aside, but, if you are already using a fork, hopefully TS is a little more buildable for your uses now with some modification; you should be able to get pure ESM out of it. Might allow some tree shaking, but, probably not too much yet.

[dsherret]

I opened denoland/deno#17862 to remove its use.

Yeah, don't worry about Deno too much because I can work around the TS codebase ok then hopefully over time can work on getting anything upstreamed. (In ts-morph though I'd rather not have to modify the ts source... I have this file with all the internal stuff I use: https://github.com/dsherret/ts-morph/blob/latest/packages/common/src/typescript/tsInternal.ts)

[jakebailey]

Please do send issues for APIs you think should be public; we already made public quite a few others in 5.0.

[bartlomieju]

Sorry for the confusion! I missed that we no longer use TSC for emitting source maps.

[dsherret]

@jakebailey thanks! If they are going away then maybe I will try again (#21021) 😄