esbuild online repl site

[hardfist]

I think something like rollup's online repl would be very convenient for playing with esbuild, which also could demonstrate the real power of using esbuild to build an unversal bundler for browser|node( webassembly support, bundle on the web, using plugin to load bare module from http & unpkg and build real web application using esbuild it self so on)。 I don't know are you intersting to do this, if you do I can do some help.

[evanw]

Yes, this is something I'm planning on doing. I already have a REPL that I use myself although it's not a bundler yet: scripts/try.html. Doing bundling in the browser would require some other features that haven't been implemented yet such as #690.

[hardfist]

it would be nice if esbuild support custom filesystem support，I currently use plugin to load files from memfs and unpkg to support bundler on browser，which is already super convenient thanks to esbuild elegant plugin system.

[zaydek]

@evanw That looks cool. Is there a Makefile or a command to try it out? I tried make platform-wasm and opening the file (no web server) but I get this: browser.js:1314 Fetch API cannot load file:///.../esbuild/npm/esbuild-wasm/esbuild.wasm. URL scheme must be "http" or "https" for CORS request.

I assumed it doesn’t use a web server because of the relative path: ../npm/esbuild-wasm/esbuild.wasm in the file.

[evanw]

Doing make platform-wasm and then opening the file is correct. But the file URL scheme is unfortunately completely broken. You have to use the http or https URL scheme via a local HTTP server. I usually use the one built into Python: python -m SimpleHTTPServer or python3 -m http.server.

[zaydek]

Thank you! That works. This is really cool.

I saw #690 -- in the future will it be possible to make this support custom plugins and talk to unpkg, etc. so you can demonstrate esbuild talking to NPM in the browser? Is that pending virtual filesystems or something else? And would NPM packages published as ES Modules work today?

[evanw]

Custom plugins already work in the browser today, but esbuild's default behavior relies on the file system so esbuild's default behavior is currently completely disabled in the browser. An in-browser esbuild REPL currently isn't able to do any bundling at all without custom plugins. The virtual file system change would make it possible for esbuild's default behavior to run in the browser too. Then you could have an in-browser REPL that shows off all of the built-in features of esbuild.

[zaydek]

That’s awesome. Looking forward to it.

This issue can probably be closed since you’ve addressed the scope of the original question. I assume you’ll make a version of the REPL available once it makes sense for you.

[evanw]

Let's keep this open until the REPL is online.

[hardfist]

@evanw @zaydek I build an simple repl site esbuild repl using esbuild-wasm as an online bundler, which supports bare import from unpkg, http import like deno, import local file from memfs, which the site itself is build&dev using esbuild, which demonstrates esbuild plugin is so flexible. The source code is here https://github.com/hardfist/neo-tools/tree/main/packages/playground, if any of you are interested.

[zaydek]

Wow. Nice work! I didn’t realize you can use unpkg to host websites.

Thank you for sharing!

[ggoodman]

@hardfist if you ever find that inter-dependencies are not resolving correctly from unpkg, you might consider throwing https://github.com/ggoodman/esbuild-plugin-velcro at the problem. It's designed to resolve graphs of modules from cdns like unpkg and JsDeliver (the latter of which I've found performs better) so that dependency ranges are respected and has some optimizations to avoid concurrent or repeated requests to the same resources.

[whatnickcodes]

@hardfist you are a legend. Great work. @evanw looking forward to your progress here. Thanks for all you do

[okikio]

@hardfist Inspired by your project, I built https://bundle.js.org it bundles js, and gives you the the final bundle size.

[curran]

Very cool that it works, but I was disappointed to learn that it loads a 9MB build.

https://github.com/hardfist/neo-tools/blob/6012ed0e605fbe5200dc47d9e670f72ecbac72fd/packages/bundler/src/lib/esbuild.ts#L9

From https://unpkg.com/browse/[email protected]/

I wonder if there is any way to get that build size down...

[evanw]

I wonder if there is any way to get that build size down...

Unfortunately there pretty much isn't, primarily because esbuild is written in Go and Go is not a language that optimizes for small executable file sizes. There are some settings that can be adjusted to reduce size, but they have already been adjusted: #836. The goal of this project was never and will never be to produce an optimal in-browser bundling experience. That wasn't even in the original design considerations at all, which is when the primary technology choices were made. Browser support has been added because people asked for it but it's still not the goal of this project. If you want a compact in-browser bundler then you should look at other projects instead.

[whatnickcodes]

100% agree. WASM version is like a cool symptom of this. Not sure of any other projects out there as eloquent as this one for in-browser bundling.

If anything that would be useful, for the people who have success with this (@hardfist, @okikio) helping expand documentation here.

Would be nice to have some dead-simple bigger examples or a dead-simple service worker example from experienced and helpful dev.

[curran]

The goal of this project was never and will never be to produce an optimal in-browser bundling experience.

Understood. That makes sense. Thanks for responding!

If you want a compact in-browser bundler then you should look at other projects instead.

Any suggestions for particular projects that might be a good fit? I'm currently using Rollup and Bublé for in-browser bundling with JSX support (implementation). This is part of an in-browser IDE used for teaching. I am looking for a modern solution that has TypeScript support to replace Bublé, which is unfortunately no longer maintained. I had high hopes for using ESBuild in place of Bublé, but the large build size is a deal breaker. I'm currently investigating Sucrase but am still not sure if it has a viable browser build. Are there any other solutions that come to mind that might work well in the browser as a replacement for Bublé? Thanks!

[okikio]

From my experience I've found that the large build size doesn't really limit you, if you preload and cache the WASM file using service-workers, plus, non of those bundler alternatives hold a candle to esbuilds performance, trust me on this, rollup, swc, etc, are all much slower than esbuild...

[curran]

trust me on this, rollup, swc, etc, are all much slower than esbuild...

Is this true even when esbuild is running in the browser? Has anyone benchmarked these for in-browser performance?

[okikio]

It wasn't a formal benchmark, but I have. For bundling a small 5Kb file it would take esbuild ~0.7s, but rollup would take 1.3s, as thing scaled up the difference become more noticeable, for bundling typescript, it takes rollup ~25-30s, when it only takes esbuild ~18s. swc is slightly faster than rollup but slower than esbuild, also, on the web swc only support synchronous API's which slows things down, the only really good option ends up being esbuild. Also, @curran the full bundle size of rollup will surprise you, check it out:

• rollup
• esbuild-wasm
• @swc/wasm-web

[EricSimons]

@evanw would you be interested in leveraging our work on WebContainers to help power the esbuild repl? We're working on similar integrations with projects like SvelteKit that rely on system level APIs via Node.js/etc as it avoids the need to diverge from how the package works on local machines.

[okikio]

@EricSimons Would it be possible to try out WebContainers for https://bundlejs.com?

[whatnickcodes]

What you all are doing with WebContainers is super exciting / creative / futuristic. Definitely think you all are onto something big.

I am unsure how I feel about this breaking the spirit of a browser based "repl" however.

[jespertheend]

I found https://hyrious.me/esbuild-repl/ which seems to work pretty nicely.

[curran]

Very cool! Looks like the wasm build in use there is only 2.4MB. Not bad!

[jespertheend]

cc @hyrious

[hyrious]

@jespertheend Thank you. In fact I do refer to the rollup repl to make the "build" part of this site.

The download size is around 2.4MB because jsdelivr enables gzip. I also made a service worker to cache each wasm response.

The upcoming part - "playground" - is one thing I'm interested in and still learning how to make it. I'd like to refer to webcontainer since normally we want to start writing a project and play with esbuild plugins, as you can see the "scripts" folder in a realworld esbuild project can be very complicated.

[okikio]

@hyrious I'd suggest taking a look at @hardfist's neo-tools project, here is a demo: https://neo-tools-git-main-hardfist.vercel.app/

[curran]

Related #1893

[karlhorky]

There is also https://esbuild.egoist.dev/ (repo: https://github.com/egoist/play-esbuild) by @egoist

...but it may be using an outdated version of esbuild, as mentioned by @g-plane here: egoist/play-esbuild#6

[okikio]

@karlhorky bundlejs is actually very similar to egoists esbuild repl, our approaches are just slightly different

[guest271314]

@evanw That looks cool. Is there a Makefile or a command to try it out? I tried make platform-wasm and opening the file (no web server) but I get this: browser.js:1314 Fetch API cannot load file:///.../esbuild/npm/esbuild-wasm/esbuild.wasm. URL scheme must be "http" or "https" for CORS request.

I assumed it doesn’t use a web server because of the relative path: ../npm/esbuild-wasm/esbuild.wasm in the file.

You can use an extension to fetch file: protocol in the extension ServiceWorker, or fetch the file on any Web site, e.g.,

manifest.json

{
  "name": "fetch-file",
  "version": "1.0",
  "manifest_version": 3,
  "permissions": [ 
    "tabs", 
    "activeTab", 
    "scripting"
  ],
  "background": {
    "service_worker": "background.js",
    "type": "module"
  },
  "host_permissions": ["file:///*", "<all_urls>"],
  "web_accessible_resources": [{
      "resources": [ "*.html", "*.js"],
      "matches": [ "<all_urls>" ],
      "extensions": [ ]
  }],
  "action": {}
}

[jakebailey]

Just to throw another hat into the ring, I throw one together today: https://jakebailey.github.io/esbuild-playground/

(It could be more efficient via the new incremental build mode, and have a better configuration system, and probably better handle module resolution, but, I needed something like this enough to warrant throwing it together.)

[guest271314]

@jakebailey Will that work to bundle all dependencies for Node.js built-in 'node:https' or Deno listenTls()?

[jakebailey]

No, this is solely for playing around with files within the window (and sharing the result in a URL), similarly to the TypeScript playground or rollup REPL.

Any external stuff will be left as-is as I've made all packages external.

[okikio]

@guest271314 bundlejs let's you set aliases, so if you alias http to https://deno.land/std/http/mod.ts it will work https://bundlejs.com/?q=(import)http&treeshake=[*+as+mod]&text="console.log(mod)"&config={"alias":{"http":"https://deno.land/std/http/mod.ts"}}

[guest271314]

@okikio Unfortunately the tab crashes on Chromium 112. What I am trying to do is bundle Node.js node or Deno deno built-in HTTPS server including all dependencies as an export to be imported into QuickJS.

I tried the instructions here https://github.com/ijustlovemath/jescx/blob/master/README.md which didn't work in my case because Node.js nightly download expects node, npm, nmx to be installed. That requires changing some paths in various files shipped in Node.js download. I generally don't install Deno, Node.js, Bun, or QuickJS local or system wide, I just fetch the latest source code and run the executable in the directory. Then esbuild threw saying I needed to set platform to node, yet that didn't work either.

I know when I do this in Deno

import { serveTls } from "https://deno.land/[email protected]/http/server.ts";

the source files are stored in /home/user/.cache/deno/deps/https/deno.land/.

I just need to compile those files, including any OpenSSL resources to a JavaScript export or shared object file that I can import into QuickJS.

My requirement specifically is to start with the required modules then build the JavaScript engine around those files. Neither Deno nor V8 provide a means to do that. deno compile compiles the entire deno executable. We wind up with a 100MB file, where after strip qjs we have a less than 1MB executable.

[guest271314]

@okikio Just tried the URL on Firefox Nightly 111, throws

SyntaxError: import.meta may only appear in a module

[okikio]

^ That's due to Firefox not supporting module workers yet, they'll add support in March

[guest271314]

Any idea why the crash on Chromium 112 (tip-of-tree Dev Channel)?

[okikio]

There is a bit of a misconception here, no matter how you try to bundle deno, if you don't bundle some sort of networking layer this will never actually work in quickjs, what you would need is https://github.com/QuickJS-web-project/quickwebserver unless I'm misunderstanding

[okikio]

Any idea why the crash on Chromium 112 (tip-of-tree Dev Channel)?

Thanks for pointing this out, I'll look into it, but it does work on Chrome 109

[jakebailey]

This all seems... very off-topic for a thread about having an esbuild REPL like rollup's. :P

[guest271314]

I'm investigating the ability to compile the same networking code to work on all of the JavaScript engines I experiment with. I tried that server. I prefer Deno server resemblance to ServiceWorker onfetch. I use QuickJS far more than Node.js or Deno, and do not want to carry around the entirety of V8 just to use the build-in server.

[guest271314]

This all seems... very off-topic for a thread about having an esbuild REPL like rollup's. :P

Well, if this is an esbuild REPL then it should be able to bundle any module from any source. That's how I would up here. On-topic from my perspective.

[okikio]

@guest271314 The problem isn't the javascript, that's easy in this situation; the problem is the C/C++ backing the network layer of the javascript. From what you've described, you want Deno's JS syntax (easy enough to do), but also don't want their Rust based network stack which is builtin with the 100 MB bundle.

I'm afraid you're left w/ 2 choices from my perspective.

1. Bundle deno's http server (bundlejs can help w/ this) and replace all their Deno specific code (e.g. Deno.listenTls, Deno.listen, Deno.unrefTimer, Deno.errors, Deno.serveHttp, Deno.readTextFileSync) with https://github.com/QuickJS-web-project/quickwebserver adapting it to work for your use case
2. You'd have to create your own server logic from scratch

You may want to look into https://workers.js.org/ while working on your solution it may help some other alternative

BTW, on Chrome Canary 102 bundlejs works, so maybe use Chrome Stable for the meantime to bundle deno

[guest271314]

@okikio Thanks. I was looking at Miniflare a couple days ago.

I am researching 1. We don't have Streams API, so we can't really upload or serve streams, e.g., duplex: 'half', new Response(new ReadableStream(...)) though others have implemented streams for QuickJS.

  /**
   * A data you want to respond with
   * @param {string|object|array|number} data
   */
  send(data) {
    this.responseType = 'string';
    const dataType = getType(data);
    switch (dataType) {
      case 'object':
      case 'array':
        this.content = JSON.stringify(data);
        break;
      case 'number':
        this.content = String(data);
        break;
      case 'string':
        this.content = data;
        break;
      default:
        this.content = null;
    }
    if (this.content === null) {
      throw new TypeError('Only "string", "object", "array" and "number" supported as response data');
    }
  }

For 2. I use window.open() with src set to a chrome-extension: URL, which becomes a WindowClient of the extension ServiceWorker, then transfer a MessagePort from the ServerWorker context to arbitrary Web pages, where I can transfer data back and forth using Transferable Streams, something like this https://github.com/guest271314/sw-transfer-stream. Among other approaches I also use WebRTC with resizable ArrayBuffers https://github.com/guest271314/offscreen-webrtc.

What I use the most is a QuickJS Native Messaging host where I dynamically set arbitrary Web sites as "externally_connectable", pass JSON arrays to the writable side of a TransformStream on the Web page https://github.com/guest271314/captureSystemAudio/blob/master/native_messaging/capture_system_audio/background.js.

Given WASI and WebAssembly and the state of the art it appears to me it should be at least possible to identify dependencies - whether they be C, C++. Rust, Zig, et al., based on the module source code itself, then build the bundle or compiled executable based on that initial and minimal or maximal requirement - instead of starting from the point of built-in included that are not intended to be used. Compilers and bundlers should be that smart circa 2023.

[evanw]

This has finally been implemented! Here it is: https://esbuild.github.io/try/. I was even able to make it work for historical esbuild versions too, all the way back to version 0.5.1.

Two forms of build options are supported: either CLI-style (example) or JS-style (example). Both are converted into a JS object that's passed to esbuild's WebAssembly API. The CLI-style argument parser is a custom one that simulates shell quoting rules, and the JS-style argument parser is also custom and parses a superset of JSON (basically JSON5 + regular expressions). So argument parsing is an approximate simulation of what happens for real but hopefully it should be close enough.

My implementation of the build API augments esbuild-wasm with a file system by pretending to be node's fs module. This means esbuild's path resolution should be a very accurate representation of the real thing. For example, you can get esbuild to read settings from package.json files that can then affect the build (here's an example).

Closing as fixed.

[ggoodman]

@evanw do you think that the WASM host setup you created is something that might warrant its own package? Was there much in the design that you found specific to your use-case or is it fairly generic?