Using Node.js, how to install R packages from Emscripten’s filesystem image with a small CRAN-like repo

[jmbo1190]

Hello,

I've been able to create a Emscripten’s filesystem image with a small CRAN-like repo with a few R packages (haven and jsonlite) and mount from the local filesystem using node.js, as shown below.
My goal is to import data using maven and return it as JSON to javascript.
But then I'm confused how to install / access R packages from that mounted filesystem ?

const { WebR } = require('webr');
const fs = require('fs');

let webR = new WebR();
await webR.init();
await webR.FS.mkdir('/localrepo');

let data = fs.readFileSync('/some/local/path/output/vfs/output.data');
let metadata = JSON.parse(fs.readFileSync('/some/local/path/output/vfs/output.js.metadata', 'utf-8'));
await webR.FS.mount(
      "WORKERFS", 
      {
         packages: [{
            blob: data,
            metadata: metadata,
       }],
      },
      '/localrepo'
   );

I can see the src and bin subfolders in the mounted filesystem image:

> res = await webR.evalR('list.files("/localrepo")')
Proxy [
  {
    obj: { type: 'character', ptr: 26422256, methods: [Array] },
    payloadType: 'ptr'
  },
  { get: [Function: get], apply: [AsyncFunction: apply] }
]
> await res.toArray()
[ 'bin', 'src' ]

I tried the following but it failed:

> await webR.installPackages(['haven', 'jsonlite'], {
...       mount: false,
...       quiet: false,
...       repos: [
...          '/localrepo'
...       ]
...    });
Warning message: 
URL /localrepo/bin/emscripten/contrib/4.4/PACKAGES.rds: Download failed. See the Javascript console for further information
Warning message: 
URL /localrepo/bin/emscripten/contrib/4.4/PACKAGES.gz: Download failed. See the Javascript console for further information
Warning message: 
URL /localrepo/bin/emscripten/contrib/4.4/PACKAGES: Download failed. See the Javascript console for further information
Warning message: 
unable to access index for repository /localrepo/bin/emscripten/contrib/4.4:
  download from '/localrepo/bin/emscripten/contrib/4.4/PACKAGES' failed
Warning message: 
Requested package haven not found in webR binary repo.

and

> res = await webR.evalR('install.packages(c("jsonlite", "haven"), repos = "file:///localrepo")')
Uncaught:
Error: Error in `utils::download.file(path, tmp, quiet = TRUE)`: download from '///localrepo/bin/emscripten/contrib/4.4/jsonlite_1.8.9.tgz' failed

Thanks a lot for your help!

[georgestagg]

Hi,

Your code to mount the filesystem image is good. However, when mounting a filesystem image you can skip the entire step of using an R package repository and installing packages. Instead, you can just mount a filesystem image that contains the files of already installed R packages, and use .libPaths() to tell R to load packages from there.

How are you currently generating your fileystem image? Are you using automation such as GitHub Actions, or manually?

---

If manually, are you using the {rwasm} package? If so, you can follow the instructions in the "Building an R package library" section of this page to build a filesystem image containing your R packages. The relevant function is rwasm::make_vfs_library().

Once you have generated that library image, it should consist of two files named something like library.data and library.js.metadata. Now, the documentation continues by starting a webserver and using webr::mount(). However, since you are in Node.js, instead copy those files into your Node project and use the code that you already have to mount them to some mountpoint (e.g. "/my-library") instead.

await webR.FS.mkdir('/my-library');

let data = fs.readFileSync('/some/local/path/library.data');
let metadata = JSON.parse(fs.readFileSync('/some/local/path/library.js.metadata', 'utf-8'));
await webR.FS.mount(
      "WORKERFS", 
      {
         packages: [{
            blob: data,
            metadata: metadata,
       }],
      },
      '/my-library'
   );

Once that's done, you can continue with something like:

await webR.evalR(`.libPaths(c(.libPaths(), "/my-library"))`)

and then finally

await webR.evalR(`library(maven)`)

[jmbo1190]

Tes I'm doing it manually.
Thanks for pointing to the section "Building an R package library", that's the step I had been missing.
And thank you for the fast answer!

[jmbo1190]

Hi, it worked 😄
I added the needed packages haven and jsonlite (+ data.table for future use) to the vfs library and was able to mount it and add it to .libPaths.

Then I started testing, and got an error message:

Error: Error in `loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]])`: there is no package called ‘rlang’

After adding 'rlang' to the vfs library and retrying, I got another error about another missing package.
I added it similarly and retried, and then again the same thing for every missing package... basically all dependencies of 'haven'.
Then I realised rwasm::add_pkg() has a parameter dependencies with default = FALSE, and setting it to NA would install all 'hard' dependencies as well.
Perhaps it would be good to modify the example on https://r-wasm.github.io/rwasm/articles/mount-fs-image.html#building-an-r-package-library-image to show that parameter, e.g.

rwasm::add_pkg("dplyr", dependencies = NA)

[jbodart-argenx]

Hello, I couldn't find the answers in the documentation, can you please clarify:

When I'm done using webR in a Node.js context, and want to properly dispose of it, is there anything else needed / recommended other than:

webR.close()

E.g. do I need to unmount filesystem images and/or directories previously mounted ?
Can I mount filesystem images and/or directories read-only (to prevent accidental modifications) ?
Can I get a list of current mounts in order to unmount them all, and not to attempt unmounting something that has already been unmounted ?

Thank you

[georgestagg]

Hi,

Nothing else is recommended other than webR.close(). My understanding of the implementation of the Emscripten virtual filesystem is that you are not required to unmount filesystem images or directories.

The only requirement I am aware of is awaiting the Promise from webR.FS.syncfs(false) when you want to persist data to local IndexedDB storage. But that only applies when you are using the IDBFS backend in a web browser.

Can I get a list of current mounts in order to unmount them all?

A convenience function to do this is not currently exposed to the main webR JavaScript thread.

I don't think it's strictly necessary, but if you are willing to experiment you could execute JS code on the worker thread using webr::eval_js() from the R session:

# Mount some filesystems
webr::mount(mountpoint = "/data", source = "[...]")
webr::mount(mountpoint = "/data2", source = "[...]")

# Find current mounts
mounts <- webr::eval_js("Module.FS.root.mount.mounts.map((m) => m.mountpoint)")
mounts
# [1] "/proc/self/fd" "/data"         "/data2"  

# Drop the first element, Emscripten handles "/proc/self/fd"
> mounts <- mounts[-1]
# [1] "/data"  "/data2"

# unmount the others
lapply(mounts, function(m) webr::unmount(m))

This will require the latest version of webR (required to return the list of strings back to R). Further documentation about the Emscripten Module.FS API can be found at https://emscripten.org/docs/api_reference/Filesystem-API.html.