Importing and Exporting in WebAssembly

[lbguilherme]

First of all, some introduction to the problem.

A Wasm Module can import and export functions and global variables. There are some other entities like tables and memories, but those aren't common and I won't focus on them.

A function signatures receive a tuple of integers/floats as arguments and returns a tuple of integers/floats as well. When importing them, they need a name and a module name. A wasm module defines many functions, some of them may be marked as exported, those must be named. Globals follow a similar logic to functions, but they are a single integer/float instead of a callable.

A compilation unit (or object file) is like a mini wasm module that the linker will work on. Its work is to merge all of them into a single wasm module. It does so in the following way:

• For every imported function from the "env" module, it finds some exported function of the same name and match them. The signature must be identical.
• Every other import from modules of some other name remain as is and will be imports on the final wasm binary.
• All exports except the ones mentioned in the linker command line are removed. After all most of them are only meant for linking of internal functions. By default a function named _start is exported if it exists.

We can look for wasi-libc as an example of this. It imports the Wasi functions from "wasi_snapshot_preview1" and exports many of the standard functions of libc. Crystal then imports those functions from "env" and the linker connects the two. Libpcre imports libc functions from "env" and exports its functions and so on.

For exporting a function on the final wasm module, its name needs to be specified as a cli flag to the linker.

Crystal currently imports only the wasi functions via wasi-libc and exports only _start, defined either with adding wasi-libc's crt1.o to the linker or by #11936. But there is need to both import and export functions on different scenarios.

A wasm program can either behave as a "command" or a "reactor".

A command program has an exported function that serve as an entrypoint. It is the only exported function and it will be called only once by the runtime. These programs starts, runs, and finishes. For Wasi this function is called _start, but it may be named differently outside Wasi.

Now reactor programs are a little different, they may export any number of functions, where one of them serves as the initialization and is invoked once by the runtime before any other. After that the other exported functions can be invoked any number of times. This is commonly used for plug-ins.

In any of these scenarios, the program may import functions provided by the runtime. Wasi-compliant runtimes (like wasmtime or wasmer) will expose the wasi standard functions, but other runtimes may expose any set of functions to be imported by the wasm module, this is also relevant for plug-ins.

This Crystal needs two functionalities to be able to fully support this:

1. Importing functions and globals from the runtime (from a module other than "env")
2. Marking some functions as explicitly exported. Those will be added to the linked flags when generating the linker command.

Importing

Rust has a wasm_import_module key on the link attribute:

#[link(wasm_import_module = "foo")]
extern {
    // …
} 

Docs: https://doc.rust-lang.org/reference/items/external-blocks.html#the-link-attribute
Original issue: rust-lang/rust#52090

I propose we use the same notation for Crystal:

@[Link(wasm_import_module: "wasi_snapshot_preview1")]
lib LibWasi
  fun random_get(buf : Int32, len : Int32) : Int32
end 

This should apply to both functions and globals defined inside the lib.

See #11935 for a possible implementation of this.

Exporting

Rust uses the following notation to mark a function as explicitely exported:

#[no_mangle]
pub extern "C" fn add(left: i32, right: i32) -> i32 {
    left + right
}

We don't have a Crystal equivalent, so I suggest adding an annotation that can be applied to fun functions, something like this:

@[WasmExport]
fun add(left : Int32, right : Int32)
  left + right
end

It would have the effect of modifying the link command for wasm. Nothing needs to change on the LLVM output.

[straight-shoota]

When compiling to native binaries, all fun methods are exported as symbols. Would that work in a similar way for WebAssembly? What's the reason for an additional export annotation?

[HertzDevil]

I think the issue is currently all top-level funs are implicitly exported, so if we simply adjust wasm's linker command it too will export everything, including e.g. our own re-implementations of compiler-rt. Thus there needs a mechanism to hide some top-level funs from the compiled object. This feature indeed is not exclusive to wasm.

There is also a parallel between those "reactor" programs and __attribute__((constructor)) / DllMain.

I would not worry about exporting names until Crystal has the more general ability to build C-compatible shared libraries.

[lbguilherme]

So, for exporting we should actually go the other way around: every fun is exported and maybe we can have an annotation or something to hide a symbol that isn't meant to be exported (like __crystal_raise for example). That makes sense.

These reactor programs are very much like shared libraries, but in the case of wasm they are still complete programs and Crystal can already target those. Hiding symbols (not exporting a function) can be viewed as a kind of size optimization now, and binary size is quite important for wasm. Either way, I'll measure it, I think the impact isn't that big yet.

Offtopic: @HertzDevil seeing now that you mentioned compiler-rt, I see that there are some functions that are not implemented for wasm and we are being forced to link against clang's compiler-rt. I'll check which of those are needed so that they can be implemented.

[orangeSi]

this had something update yet? If export funciton works, people will can use crystal to generate wasm to use in python code, mean use python + crystal + wasm instead of python + C for some task!

This feature may attract more people to use crystal in other languages( example:Python).

[lbguilherme]

Bump.

There is a PR #11935 implementing this proposal.

Import

The proposal is aligned with other languages and fits well with the lib syntax. Most external libs don't need this feature to import, but Asyncify can only be imported this way and this is a blocker to implement Fibers and the GC. With the GC enabled Crystal will already be usable on real world scenarios.

Export

Exporting functions allows the embedder to call Crystal methods from outside, like exposing a WASM module to a Web page with JavaScript. The original proposal was to add a custom annotation to mark which fun functions to export. Later I proposed something simpler: to export all fun functions (this is what is implemented on the PR) and perhaps later add some annotation to not export something. This is already the behavior of the other architectures (fun produces a public external symbol on the final executable`).

As a proof of concept I implemented exporting methods in my crystal-js shard:

require "js"

JS.export def print_hello(name : String)
  puts "Hello, #{name}!"
end

And the shard will generate a low level fun with the appropriate handling code to convert and transfer a String between JS and Crystal. It currently uses some linker flag hacks to get this working, but those can be removed once this PR is merged.

---

I would like some direction from the core team about this: It would be better to wait for more user demand before going forward? Or is there need for more detailed discussion before committing on a particular implementation? Or there are other priorities right now and this should just wait a little bit more? I'm good with any of these.

[orangeSi]

Hello! @straight-shoota @HertzDevil approve this PR #11935 need more discussion among you and @lbguilherme or someone? This PR #11935 had been not active from April.

I just want Crystal give more support for Webassembly...

And the people which contribute mostly to Webassembly maybe @lbguilherme? If ture， how about you invite @lbguilherme to be one of Crystal Lang Team Core Member? That maybe will speed up the support of Webassembly...

[lbguilherme]

Closing as #11935 was merged 🚀