Testing strategy for wasm32-unknown-unknown

[alexcrichton]

One thing that came up in a conversation with @ashleygwilliams and @lukewagner today is: How do we test wasm code?

This is a pretty broad sentiment but the question was born out of the idea that we eventually want to verify #[wasm_bindgen] annotations are correct and/or you're importing the right packages and such. Right now we actually don't have a great way to test this! Most usages of #[wasm_bindgen] have been demos and/or applications which have testable end products, but we really want testable intermediate products (libraries) as well to ensure everything works.

So how do we actually expect users to test their wasm code? How is #[wasm_bindgen] tested, for example? I think personally we want to shoot for the answer to this question being simply:

$ cargo test --target wasm32-unknown-unknown

But... we've got a long way to go to get there! In the meantime here's a possible strawman which could work:

• There's some package on crates.io you install which installs something like wasm-test-runner. This could be included in either the wasm-pack or wasm-bindgen repos, for example.
• You globally configure Cargo with target.wasm32-unknown-unknown.runner = 'wasm-test-runner'. This means that cargo test --target wasm32-unknown-unknown will always run through that test runner, so it basically passes off tests to that harness to get executed.
• The test runner will automatically execute wasm-bindgen, the CLI, if necessary. For now it'd probably require the --nodejs flag.
• The test runner will automatically execute wasm-pack, the CLI, if necessary
• The test runner will automatically run npm install, if necessary
• Finally, the test runner would run node by importing the JS shim wasm-bindgen generated, and after that's done it executes the main function.

I think with a strategy like that we have something which looks like "write tests as you normally would in Rust and they'll naturally get executed on the wasm target". The next parts, however, are tricky. Things left to be implemented are:

• Somehow we need to hook println! and stdio. Otherwise prints and panics will simply be shown as unreachable with no source information at all (eew!).
• Somehow we need to catch failing tests and not abort the whole process. Tests should continue to execute afterwards.
• Tests ideally are still executed concurrently.

---

That's sort of a semi-strawman, but it's certainly a lot of work and not going to solve this issue in its entirety! This likely means that we, as a working group, will want to follow the custom test framework RFC pretty closely to make sure it accommodates what we'd need as well.

In any case, curious to hear what others' thoughts on this are!

[steveklabnik]

This is basically exactly what I wanted for this feature, so big 👍 to me.

The only thing would be small details; like it'd be very cool if you didn't need special configuration. Big picture wise, 💯 though

[Pauan]

With stdweb it's currently possible to write unit tests which are then executed in either the browser or node.js, however one thing that we've really wanted is the ability to write asynchronous unit tests, such as with Futures or Promises.

So that's another important use case, since Promises and async APIs are so common in JS. Ideally it should be possible to just use #[async] and #[test] together and have everything Just Work, but right now it doesn't.

[fitzgen]

I like this a lot, and its the direction I think we were already moving with the custom test harness eRFC. I was expecting to run in a headless browser rather than node, but either way depending on what environment you're targeting.

I have some uncertainties about exactly how the harness (and compiling the harness) will work, since it needs to be compiled for the host, but work with cross-compiled w32-u-u artifacts.

[fitzgen]

https://crates.io/crates/fantoccini looks pretty neat. A WebDriver library for controlling/automating testing with (potentially headless) browsers.

[Hywan]

The strategy I use for the WASM binding of the Gutenberg parser is to use an ECMAScript Module (ES Module) for the boundary layer (reading/writing memory of a WebAssembly module instance, and exposing an API to the JS environment). An ES Module can be used in a browser, and also in NodeJS (with the --experimental-modules option). And in my specific usecase, I have written an executable to parse data and emit JSON or debug output. But the trick is that this approach allows to run the boundary layer/decoder/encoder within NodeJS, so outside a browser context, but the code is shared between both environments, it's a way to test the browser's behavior directly.

It's a specific and closed/delimited usecase, but I think using ES Module is an interesting approach.

[alexcrichton]

I've started to develop this for wasm-bindgen's own test suite in rustwasm/wasm-bindgen#524, and so far I'm pretty happy with the result (it's basically the OP). I've also written a small overiew about it.

It's not clear to me yet where headless testing fits into this sort of strategy nor whether the JS integration there is good enough to use in practice. If anyone's got thoughts though it'd be most appreicated!

[alexcrichton]

I'm gonna close this in favor of wasm_bindgen_test and wasm-pack test which are both in development, which solves at least what I was hoping for with this!

[autodidaddict]

@alexcrichton what is the strategy for people writing code targeting wasm32-unknown-unknown that want to unit test their code, and they're not using wasm bindgen or wasm pack? I can't run cargo test on my library

[fitzgen]

You can write a custom test runner with a wasm engine and target.<triple>.runner in .cargo/config.toml or via the CARGO_TARGET_<triple>_RUNNER env variable. This is what wasm-bindgen's and cargo-wasi's test runners are.

https://doc.rust-lang.org/cargo/reference/config.html#targettriplerunner

[autodidaddict]

I'd like to avoid having to write a "custom test runner", especially when my unit tests are exercising code inside the module. If I want to acceptance test, then I'll use my engine and invoke functions that cross the host-guest boundary.

I guess to me writing an engine-based runner that crosses the wasm module boundary feels like an acceptance test. I'd actually like it to be able to run the unit tests almost like it was just a regular Rust library.

[alexcrichton]

The strategy is what @fitzgen mentioned. The custom test runner is used to actually execute the WebAssembly binary, it is not linked in to provide a different test framework. By default Rust code using cargo test, even on wasm, uses libtest in the standard distribution.