WasmFS Documentation

[tlively]

WasmFS should be well documented, with docs covering this non-exhaustive list of topics:

• How to build with WasmFS
• How to configure various backends and reason about when caches, etc. would be helpful.
• How to write your own backend.

[curiousdannii]

My project has two things that I'd like to use WasmFS for, so it would be great if the docs explained both of these:

1. A custom async FS. I have a custom FS currently, but it's hampered by not being async
2. Async stdin, which I currently implement by wrapping getc with a function that calls (if it's getting stdin) an EM_ASYNC_JS function. I'm lucky that the project I'm porting only accesses stdin via getc, as it would be messy to have to wrap lots of other functions too.

Also, enabling and mounting FSes used to happen in JS code. Will that still be the case? Or will we be able to put this code into the wasm. My port doesn't modify main, so it would be nice if this doesn't require that either...

this.EFS = new EmglkenFS(this)
Module.FS.mkdir('/emglken')
Module.FS.mount(this.EFS, {}, '/emglken')
Module.FS.chdir('/emglken')

[tlively]

re: mounting, I expect we will have both C and JS APIs for that. The story for Async APIs hasn't quite been figured out, but that's tracked in #15964.

[curiousdannii]

@tlively #15964 is about a new async API, right? I was expecting that it would possible to make the standard posix FS API async with Asyncify.

[tlively]

Oh gotcha. Yes, we might have to do something to get Asyncify working as expected as well. (Or maybe it will Just Work, not sure). I filed #16053 to track that separately.

[nachoab]

Is there some example of compiling including OPFS? Setting WASMFS=1 doesn't seem to include it, even though it looks like it is. Also, using current FS you have to use the mount method to specify a different FS provider, does WasmFS use this same API?

[mere-human]

Is there some example of compiling including OPFS?

@nachoab I think the best example would be tests, for instance https://github.com/emscripten-core/emscripten/blob/main/test/wasmfs/wasmfs_opfs.c

[curiousdannii]

It's taken a long while, but I'm ready to start working on my own WasmFS backend.

How "ready" is WasmFS now? I know some people are using it, but I also see lots of open issues and tickets in the project/milestone. It seems like there's still not really any docs, we just have to learn from the other implementations?

I suspect it might not be that difficult to use WasmFS with Asyncify, because you can put anything through Asyncify. The question is more about what's efficient and ergonomic.

An alternative idea I had was to somehow replace stdio.h with a custom implementation, but that would probably be much more complex.

[tlively]

We have some very large applications using WasmFS, especially with the OPFS backend, so it's "ready" for production use in the sense that it is stable and correct enough. We haven't been able to implement as many new features as we wanted, though, such as other concrete backends (for IndexedDB especially) and virtual backends for caching and layering, etc. We also haven't been able to finish all the compatibility work that we want to have done to allow us to turn on WasmFS by default.

[art926]

Could you update
https://emscripten.org/docs/api_reference/Filesystem-API.html#file-systems
to mention OPFS, please?) We're very interested in using it.

[adamziel]

We have some very large applications using WasmFS, especially with the OPFS backend, so it's "ready" for production use in the sense that it is stable and correct enough.

@tlively I would love to explore using the OPFS backend in WordPress Playground, but I'm not sure how to do it. Would that work without pthreads support? A perfect option for Playground would be an asynchronous filesystem backend as we don't support pthreads. We're using MEMFS in web browsers and MEMFS+NODEFS in Node.js. Would you be able to point me to any resources, or write up a guide in a comment in this issue?

[jozefchutka]

I also came here to explore options for OPFS backend. Preferably would like to mount FileSystemDirectoryHandle as mentioned in #15964 (comment) but have no idea how/where to start. Will OPFS backend be provided by emscripten or something a dev would implement himself following some documentation?

[adamziel]

Looping in @ThomasTheDane @tomayac – an OPFS backend would be a highly useful addition.

[kripken]

I would love to explore using the OPFS backend in WordPress Playground, but I'm not sure how to do it. Would that work without pthreads support?

I believe the OPFS backend requires threads or JS Promise Integration (JSPI). One of the two is needed so that we can handle the async behavior of the JS APIs there. Here are the tests that we run:

emscripten/test/test_browser.py

Lines 5205 to 5225 in a756ff0

	@no_firefox('no OPFS support yet')
	@no_wasm64()
	@parameterized({
	'': (['-pthread', '-sPROXY_TO_PTHREAD'],),
	'jspi': (['-Wno-experimental', '-sASYNCIFY=2'],),
	'jspi_wasm_bigint': (['-Wno-experimental', '-sASYNCIFY=2', '-sWASM_BIGINT'],),
	})
	def test_wasmfs_opfs(self, args):
	if '-sASYNCIFY=2' in args:
	self.require_jspi()
	test = test_file('wasmfs/wasmfs_opfs.c')
	args = ['-sWASMFS', '-O3'] + args
	self.btest_exit(test, args=args + ['-DWASMFS_SETUP'])
	self.btest_exit(test, args=args + ['-DWASMFS_RESUME'])
	@no_firefox('no OPFS support yet')
	def test_wasmfs_opfs_errors(self):
	test = test_file('wasmfs/wasmfs_opfs_errors.c')
	postjs = test_file('wasmfs/wasmfs_opfs_errors_post.js')
	args = ['-sWASMFS', '-pthread', '-sPROXY_TO_PTHREAD', '--post-js', postjs]
	self.btest(test, args=args, expected='0')

The code in those tests could be an easy way to get started with the OPFS backend.

Separately, it would be good to update the docs on the website.

[adamziel]

Thanks @kripken! I started exploring and got some initial successes, I'll report back when I have something workable. It will probably take a few months of taking a few moments here and there to get this right – just like supporting JSPI in Playground took 1,5 years. It wasn't an easy process.

It would be much easier to use this API if there was a high quality documentation page. Not blog posts scattered on v8.dev and brief descriptions on https://emscripten.org/docs/, but an actual solid learning resource with detailed, step by step guides, code examples, descriptions of different wasmfs backends, specific terminal commands to run, outcomes to expect, mental models explained, and live demos of the built binaries. Also, a transition guide from the JavaScript FS to WASMFS would help a lot.

Right now we, the developers trying to use WebAssembly, are in a tough spot. It's very unclear what's in progress, what's finished, what's the browser support, how to use what, and which APIs are going to change in two versions.

[kripken]

@adamziel I agree, this could be much improved. There is also the remaining work to get WasmFS enabled by default, and I was thinking that might happen before, but perhaps it doesn't need to be in that order. I'm not sure when I'll have time myself to get to either, but I'd like to (and others might get to it before).

fyi @tomayac , this could be a promising area for docs or a blogpost or both.

[tomayac]

I agree that there's a need to document this properly. Meanwhile, this repository courtesy of @milky2028 might help people bootstrap their apps. May I also recommend people working on this install my OPFS Explorer Chrome DevTools extension? The files that you open with this demo are persisted across reloads, as you can verify with the extension.