Add filesystem interface for the File System Access API #16804

connorjclark · 2022-04-24T23:59:07Z

This patch adds a new filesystem interface FSFS which utilizes the new File System Access API. Like idb, this is an asynchronous API. The implementation is nearly 1:1 with the idb interface, with a few key differences:

mode is not supported. All files and folders are written to Emscripten's FS as 0777
A directory handle must be provided with the proper permissions granted to it. It's up to the caller to handle that. A real application would also store the handle locally in idb for later retrieval, however all that seemed out of scope (users would want full control over the permission and caching flow anyhow)

Closes #15277, #12428

connorjclark · 2022-04-25T00:00:15Z

src/library_fsfs.js

+  },
+  $FSFS: {
+    DIR_MODE: Number("{{{ cDefine('S_IFDIR') }}}") | 511 /* 0777 */,
+    FILE_MODE: Number("{{{ cDefine('S_IFREG') }}}") | 511 /* 0777 */,


This looks a little funny, but it's wrapped in a string because otherwise my editor barfs on parsing the rest of the file, and it was hard to work like that.

connorjclark · 2022-04-25T00:03:44Z

src/library_fsfs.js

+      Object.keys(src.entries).forEach(function (key) {
+        const e = src.entries[key];
+        const e2 = dst.entries[key];
+        if (!e2 || (FS.isFile(e.mode) && e['timestamp'].getTime() > e2['timestamp'].getTime())) {


This line is peculiar, especially when you note that the idb interface does a simple equality check. The problem here is the the FS API does not allow you to set the mtime directly, it's just whatever the time was when it was written (as expected). So the modification time when written to disk will have to be ~syncfs time, which is strictly greater than the emscripten FS's mtime.

There's probably a race condition here, perhaps "syncfs" should be wrapped with some promise mechanism that prevents concurrent syncs? I'm not sure–I've ignored it because it can be avoided entirely if you just don't do concurrent syncs.

connorjclark · 2022-04-25T00:06:25Z

site/source/docs/api_reference/Filesystem-API.rst

@@ -57,7 +57,7 @@ However, due to JavaScript's event-driven nature, most *persistent* storage opti
 File systems
 ============

-.. note:: Only the :ref:`MEMFS <filesystem-api-memfs>` filesystem is included by default. All others must be enabled explicitly, using ``-lnodefs.js`` (:ref:`NODEFS <filesystem-api-nodefs>`), ``-lidbfs.js`` (:ref:`IDBFS <filesystem-api-idbfs>`), ``-lworkerfs.js`` (:ref:`WORKERFS <filesystem-api-workerfs>`), or ``-lproxyfs.js`` (:ref:`PROXYFS <filesystem-api-proxyfs>`).
+.. note:: Only the :ref:`MEMFS <filesystem-api-memfs>` filesystem is included by default. All others must be enabled explicitly, using ``-lnodefs.js`` (:ref:`NODEFS <filesystem-api-nodefs>`), ``-lidbfs.js`` (:ref:`IDBFS <filesystem-api-idbfs>`), ``-lfsfs.js`` (:ref:`FSFS <filesystem-api-fsfs>`), ``-lworkerfs.js`` (:ref:`WORKERFS <filesystem-api-workerfs>`), or ``-lproxyfs.js`` (:ref:`PROXYFS <filesystem-api-proxyfs>`).


Note: I couldn't find any documentation on how to build and view these docs, so I haven't verified these changes.

tomayac · 2022-04-25T07:47:42Z

site/source/docs/api_reference/Filesystem-API.rst

+
+The *FSFS* file system implements the :js:func:`FS.syncfs` interface, which when called will persist any operations to the attached ``FileSystemDirectoryHandle``.
+
+This uses the `File System Access API <https://web.dev/file-system-access/>`_, which is currently only supported by Chromium browsers.


IIUC this would also work with the Origin Private File System (OPFS) as long as you pass it a root FileSystemDirectoryHandle obtained via navigator.storage.getDirectory(), which is also supported by Safari!

The OPFS is also mentioned in #15277 and #12428.

That's my understanding too...I'll verify and update the text.

Yup, this works too. And it allows the test to run without any user interaction, which is great!

kripken · 2022-04-27T22:22:23Z

It definitely makes sense to support this browser API, but we are hoping to deprecate the old FS soon given the progress on WasmFS. Bugfixes are great there, of course, but for new backends it probably makes more sense in WasmFS, though things are not 100% stable there (but there is a PR for an OPFS backend, #16813). @tlively what do you think?

connorjclark · 2022-04-27T23:08:32Z

You probably know this already, but just to be clear: There's a definite performance benefit to the synchronous FS API, but if you can't run that code in a worker for some reason then it isn't an option (I'm not 100% sure if my application could even make use of a worker version of the FS). That's the biggest difference between these two implementations, the other being not using the new WasmFS backend. I'm not familiar with WasmFS yet so maybe this is not a real problem.

Oh, and this PR accepts any directory handle, while the one you shared just uses the private directory (I've commented on the PR about that, ~~it should be OK to make it more generic and allow any directory handle~~).

Any chance this could land, even if it's just around for a couple releases? Sounds like WasmFS is an unknown amount of time away from being stable... but if it's just a matter of days then yeah I don't see the point. Although I imagine the old FS code will stick around for X releases just in case?

If not, no worries. I've got it working locally for my application, just figured it'd be good to upstream it.

kripken · 2022-05-02T18:33:57Z

Any chance this could land, even if it's just around for a couple releases? Sounds like WasmFS is an unknown amount of time away from being stable... but if it's just a matter of days then yeah I don't see the point.

If this PR is urgent to land then I wouldn't oppose it, as it doesn't add much complexity or other risk. However, if this can wait say a few months then I'd recommend working on a WasmFS version for it instead.

I imagine the old FS code will stick around for X releases just in case?

Yes, even after we enable WasmFS by default I expect the old FS will stick around for a while.

connorjclark · 2022-05-02T18:59:05Z

Not urgent for me. @tomayac are you aware of any emscripten users that would benefit from this today?

This snippet should also serve as a drop-in library, it's basically what I'm using right now: https://gist.github.com/connorjclark/4707adbb5f2ea2322511b7e43942b10b

Depending on when WasmFS is ready, I may not be readily available to work on the FS API backend, but I'll give it a shot if I've got the time. Is there a good issue I can follow to know when it'd be time to try?

tlively · 2022-05-02T20:15:38Z

@connorjclark, actually, it might be reasonable to try adding this support as soon as #16813 lands if you want to build the new functionality into that OPFS backend.

connorjclark · 2022-05-02T20:26:47Z

I'm not totally certain how much/if any of that can be reused for the generic FS API given the usage of createSyncAccessHandle, but I haven't looked closely yet.

tlively · 2022-05-02T20:30:06Z

I think if you add a bool to OPFSFile (and rename the classes, I guess) saying whether the file is in OPFS or not, then you should be able to pass that out to the JS to get the non-sync handle as necessary.

connorjclark · 2022-05-02T20:33:50Z

Yeah, renaming + offering a way to pass in your own (non-sync) directory handle (and using the sync navigator.getDirectory() by default if not provided) might not be a bad way to combine all this into one backend. I'll experiment when you've landed it!

tomayac · 2022-05-03T08:02:31Z

Not urgent for me. @tomayac are you aware of any emscripten users that would benefit from this today?

I think a couple of folks would be pretty excited, but admittedly from the people I know @ RReverser (mention broken on purpose) is closer to the scene, but (i) has left Google and (ii) has other things on his mind at the moment.

Jaifroid · 2023-01-08T19:22:30Z

@connorjclark Because passing ArrayBufers to WORKERFS appears to involve copying or serializing of data on Android (not on Linux or Windows, where just the reference is passed), over at Kiwix we're looking for a way to work more directly with JavaScript File handles (whether the good old File API or the File System Access API).

Would this code allow access to a 92GB file handle that has been picked by the user? I know the File System Access API file and directory pickers aren't supported on Android (Chromium), and they're not supported at all on Firefox (on any platform), but would this code work with File API as a fallback?

I'll make a separate issue about our problems with WORKERFS on Android. It works great on other platforms, even with huge archives, but only works with small archives (probably less than system memory size) on Android, which is not supposed to happen.

connorjclark · 2023-06-15T17:26:06Z

I don't currently plan on re-adapting this PR to use the other backened any time soon, so I'll close this. On the other hand, if it's decided this current implementation is ok as-is, I'm happy to rebaseline.

I have a drop-in replacement for this patch available here, for anyone interested: https://github.com/ArmageddonGames/ZQuestClassic/blob/main/web/fsfs.js .

If someone ends up using the above and finds issues or makes useful improvements, I'd be grateful if you upstreamed them to me (via the above repro would be more appropriate than in this PR thread - https://github.com/connorjclark/emscripten/tree/fsfs is also ok if more convenient).

connorjclark commented Apr 25, 2022

View reviewed changes

tomayac reviewed Apr 25, 2022

View reviewed changes

connorjclark force-pushed the fsfs branch from 148283e to 524cb4b Compare April 26, 2022 02:13

Add FSFS filesystem interface

26787ac

connorjclark force-pushed the fsfs branch from 524cb4b to 26787ac Compare April 26, 2022 03:47

ryanking13 mentioned this pull request Jun 8, 2022

Native file system API pyodide/pyodide#738

Closed

ryanking13 mentioned this pull request Aug 18, 2022

ENH Add API for mounting native file system pyodide/pyodide#2987

Merged

3 tasks

ryanking13 mentioned this pull request May 28, 2023

Using pyodide.mountNativeFS and syncfs deleted multiple files unexpectedly pyodide/pyodide#3881

Open

connorjclark closed this Jun 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add filesystem interface for the File System Access API #16804

Add filesystem interface for the File System Access API #16804

connorjclark commented Apr 24, 2022 •

edited

Loading

connorjclark Apr 25, 2022

connorjclark Apr 25, 2022

connorjclark Apr 25, 2022

tomayac Apr 25, 2022

connorjclark Apr 25, 2022

connorjclark Apr 26, 2022

kripken commented Apr 27, 2022

connorjclark commented Apr 27, 2022 •

edited

Loading

kripken commented May 2, 2022

connorjclark commented May 2, 2022

tlively commented May 2, 2022

connorjclark commented May 2, 2022

tlively commented May 2, 2022

connorjclark commented May 2, 2022 •

edited

Loading

tomayac commented May 3, 2022

Jaifroid commented Jan 8, 2023

connorjclark commented Jun 15, 2023 •

edited

Loading


		The FSFS file system implements the :js:func:`FS.syncfs` interface, which when called will persist any operations to the attached ``FileSystemDirectoryHandle``.

		This uses the `File System Access API <https://web.dev/file-system-access/>`_, which is currently only supported by Chromium browsers.

Add filesystem interface for the File System Access API #16804

Add filesystem interface for the File System Access API #16804

Conversation

connorjclark commented Apr 24, 2022 • edited Loading

connorjclark Apr 25, 2022

Choose a reason for hiding this comment

connorjclark Apr 25, 2022

Choose a reason for hiding this comment

connorjclark Apr 25, 2022

Choose a reason for hiding this comment

tomayac Apr 25, 2022

Choose a reason for hiding this comment

connorjclark Apr 25, 2022

Choose a reason for hiding this comment

connorjclark Apr 26, 2022

Choose a reason for hiding this comment

kripken commented Apr 27, 2022

connorjclark commented Apr 27, 2022 • edited Loading

kripken commented May 2, 2022

connorjclark commented May 2, 2022

tlively commented May 2, 2022

connorjclark commented May 2, 2022

tlively commented May 2, 2022

connorjclark commented May 2, 2022 • edited Loading

tomayac commented May 3, 2022

Jaifroid commented Jan 8, 2023

connorjclark commented Jun 15, 2023 • edited Loading

connorjclark commented Apr 24, 2022 •

edited

Loading

connorjclark commented Apr 27, 2022 •

edited

Loading

connorjclark commented May 2, 2022 •

edited

Loading

connorjclark commented Jun 15, 2023 •

edited

Loading