specifying the C/C++ ABI (but maybe not in v.1)

[lukewagner]

I've been assuming we'd leave ABI up to compilers, but while thinking through the motivations for specifying a text format in #65, I realized the same line of reasoning applies to ABIs: basically, if there is some convention that everyone will be forced to choose from day 1 (in the case of #65, text projection), then we run the serious risk of having multiple diverging conventions appear which unnecessarily hurts usability of the ecosystem. In the case of ABIs, there have always been multiple ABIs in C/C++, so it's easy to take for granted that ABIs are just an essential annoyance. However, having multiple ABIs is just part of the inherited history of FORTRAN->C->C++ (and the reuse of shared linkers). With WebAssembly, we necessarily have a clean break in ABI, so we have an opportunity.

A question is whether we can safely delay specifying the ABI until after v.1. Since dynamic linking will put the ABI issue front and center, maybe we can just wait to specify the ABI until then. However, I was thinking that even static link libraries might lead to multiple competing ad hoc ABIs. In fact, with the same motivation as ABIs, I was thinking we should specify static link library container formats. This could make downloading and using a library in C++ (almost, headers) as easy as it is with some other languages. In some sense, these improved ergonomics come to us cheap, prepaid by WebAssembly being a portable assembly language.

I realize (and fully embrace) we're trying to be minimal in v.1, but, unlike most of the items we're able to punt to later versions, there is a real, long-term opportunity cost to punting if it leads to fragmentation. That being said, I'd be happy to delay this decision until "late" in v.1, after we have a lot more experience and understanding of the library situation.

Thoughts?

[jfbastien]

I'm not sure that we want to specify ABI for static libraries: that would be made redundant when C++ modules are standardized (I'm still hoping in C++17).

I'm very happy to wait until dynamic linking to specify ABI.

I think there will be multiple parts to this, the most important being:

• How does a function signature get represented:
  • Which types exist (only basic, or structs/arrays/unions too?)
  • What do we do with multiple returns?
  • How do varargs work? I'd just disallow them, and in C/C++ expand them.
  • Byval struct arguments: do we split them up, or pass them on the stack by pointer?
  • How does the stack work? Alignment?
• How does its name get mangled? Do we include type information (I think we should!)?

The same applies for globals as for functions.

[lukewagner]

I haven't actually been following C++ modules too closely, so sorry if this is naive, but since the C++ modules proposals operate at the source/semantics level, how would a standard at the binary and ABI level be made redundant? If anything, it seems like these two things would be complementary.

[jfbastien]

I was just talking about static libraries. I'm thinking that it would be nice to be able to put multiple libraries together into one final executable, and perform LTO (or some form of thin LTO) on the lot of them. Having a serialized AST is pretty convenient for that, and IIUC modules should enable this.

[sunfishcode]

For the "C++ ABI" part, we're currently using the Itanium C++ ABI plus a few tweaks. The Itanium C++ ABI is well regarded, so I imagine we can just specify that. I wouldn't want to document the entire Itanium ABI details ourselves, but specifying that it is Itanium, and what the tweaks are, would be reasonable.

For the "C ABI" part, documenting things like the sizes and alignments of all the primitive types is a traditional thing to do, and isn't hard. Also, documenting what happens with these types in arguments and return values is useful, since lots of things like to interoperate at the "C ABI" level. The main question for us there is what we're going to do for multiple return values, if anything. I'm not super excited about specifying the details of struct layout or bitfield rules and such, and might be inclined to just say something along the lines of "it's obvious in obvious cases, but see what clang does for the full details".

For the libc and C++ standard library ABI parts, we're probably just going to compile whatever library we pick, and then point to the result and say "that's the ABI".

For the LLVM IR part, I'm hoping that we can manipulate the target triple in a way which will give us some ability to version the ABI, so that we can make adjustments, especially early on.

For the binary container format part, I don't think we've agreed on what this will look like yet. For something containing code which has a C/C++ ABI, it would be nice if we could ensure that it has a versioning mechanism so that we can detect ABI mismatches. I know this is contrary to the "versionless" goal of the platform, but this is a level above the platform, and ABIs are tricky enough that I think the flexibility is an important consideration.

[sunfishcode]

JF mentions several things above that I missed too:

• how does the stack work: Yes, we should definitely document this from the start.
• varargs: It's traditional to include how these are lowered in a "C ABI". Expanding them as JF describes is simple and effective. We should probably document how they're expanded at some point, but I wouldn't consider it high priority.
• byval struct arguments: clang has a default lowering for these. I don't actually know what it is offhand. It's not uncommon to document such things for a platform, but I wouldn't consider it high priority though.

I think overall, the strategy for C/C++ ABIs is:

• just use clang's defaults and best practices whenever they make sense
• strive to keep the ABI clean and free of historical baggage
• document the ABI incrementally, starting with just the basics that are most often needed
• version everything, and plan to throw a few ABIs away while we find our way around

[kripken]

Sorry for my confusion, but are we discussing 'speccing' the ABI, as in similar to the spec for .wasm (which browsers will implement to the letter), or are we discussing 'specifying' as in, we'll document the ABI that is used in the LLVM backend we are writing, for convenience and to avoid fragmentation (but other compilers might emit totally other stuff, and would still run in browsers, and that would be fine)?

(The ambiguity would not have been present if this were filed in the llvm repo, but given it is in the spec, I'd like to be sure I follow this.)

[sunfishcode]

I assume we're talking about the latter here.

[kripken]

Cool, thanks.

[lukewagner]

@jfbastien I don't follow your LTO/modules point. When you say "serialized AST" are you referring to a technique for how compilers will implement C++-modules or WebAssembly? If the former, would it be standardized/interoperable?

@kripken Yes, the latter, similar in motivation to the text format. I had even thought that we could segregate both the text format and ABI into a separate section or document from the "core" semantics. The important thing is to avoid fragmentation, I think.

@sunfishcode It's fine if the ABI we specify is "what clang does with tweaks" so long as other compiler teams (e.g., MSVC) have an opportunity to comment such that they would not later want to have a separate ABI.

[jfbastien]

Serialized AST is for modules. It wouldn't be standardized or interoperable IIUC.

[lukewagner]

@jfbastien Ok, would this serialized AST be distributed or would it always be created locally as a temporary file? If the former, would it be portable even between different versions of the compiler?

[kripken]

Another option for LTO might be "fat object files", containing both LLVM bitcode and compiled wasm. When building a concrete shared library, the compiled wasm would be combined and only it retained; when building a concrete final executable, the LLVM bitcode could be combined and fully LTO'd. It seems like this would allow for both full-speed incremental compilation as well as full-power LTO, at the only cost of some disk space and access time (not computation time, as the LLVM bitcode would have existed anyhow - it just wouldn't be thrown away).

(The LLVM bitcode would obviously not be standardized, nor portable between compiler versions, but seems to be the only way to ensure full-power LTO.)

I'm curious how this compares to a serialized AST approach?

[jfbastien]

@lukewagner C++ module serialization aren't specified at this time. Clang simply serializes its AST, which changes from version to version and is entirely different from other compiler's ASTs. I just discussed this with some goog folks who work on clang and they agree that at some point it may get standardized, but at this point in time the goal is for modules to help for modularity and compilation speed. Sharing between compilers can be added at a later time, but that would definitely be after C++17 (it's not even clear modules themselves will make C++17).

I'm not sure I like @kripken's suggestion for fat object files. That seems pretty different from what LLVM usually does (and we've learned that being too different is a bad idea!). It will also probably cause issues when IR and wasm are slightly different.

[kripken]

I agree it's different, but I think there are a few areas where we can be a little different where it helps: For example, having a single ABI instead of several competing ones is a good difference from the mainstream C++ ecosystem :) And having a secure stack is another, etc. etc.

What the fat objects idea tries to achieve is to make LTO - at least LTO for code size reduction - a standard feature, and easily used, which matters for the web more than for any other platform. For the same reasons as we are working to ensure the binary format is as small and compressible as possible, and that browsers only support the binary format, we should have the toolchain be able to easily emit LTO'd code. (I would also argue it should emit LTO'd code by default in optimized builds, but that's a separate matter.)

If we don't get this right early on, then we'll end up like the rest of the C++ ecosystem: LTO is something that few projects benefit from, and a lot of unneeded code will be transmitted on the web. Nicely compressed, but still lots of unneeded bits being downloaded.

Of course, I would 100% agree that if fat objects prevent incremental compilation, or slow down compilation in a noticeable way, then the idea would be bad. However, since all this idea does is save the LLVM IR we already have, I think it will have negligible overhead.

You mentioned a concern regarding IR and wasm being different. Can you please elaborate - I don't follow?

[jfbastien]

I think we'd want to make this a standard feature through LLVM in general, not one that's specific to wasm. The weirder a wasm toolchain is, the less likely we are to get upstream LLVM's support.

Compile time is also pretty painful with PNaCl/Emscripten, so being similar to other C++ toolchains is desirable in that sense. I do agree that LTO gives us desirable benefits on size and speed, and that we probably want to support both LTO and non-LTO.

What I meant by IR and wasm being different: IR -> wasm will lose some information, so doing LTO with IR versus non-LTO with wasm using the same fat object files can produce different results. That's somewhat un-intuitive, though it's a general problem with LTO that we're compounding by adding more differences.

[kripken]

Compile time is also pretty painful with PNaCl/Emscripten, so being similar to other C++ toolchains is desirable in that sense.

I agree 100%, which is why, as I said, I believe this proposal is similar to other C++ toolchains that way. It would be a huge improvement over current PNaCl/Emscripten. Again, this proposal should add only negligible overhead - it just saves the LLVM IR, instead of throwing it in the trash. That's all! :)

doing LTO with IR versus non-LTO with wasm using the same fat object files can produce different results

Aside from an LTO bug, I don't see how? (Yes, wasm loses information. But it - or the container format it is inside - retains everything you need for linking purposes - it has to, if we are considering using it as an object file format. Aside from that, all the information that was lost is information that cannot affect semantics.)

[lukewagner]

I was hoping we could avoid using LLVM IR for LTO purposes by instead generating the LLVM IR from wasm directly. While some information is lost in the LLVM IR -> wasm lowering, it seems like:

• for many of the use cases (DCE, inling), the lost information wouldn't matter
• for the cases we determined (with measurements) mattered, we could either:
  • include it in the wasm format (it enables better codegen, why lose the information)
  • include it as a separate "LTO" module section which could be much smaller than full LLVM IR, standardized, and stripped when building a release module (although it would be innocuous if it wasn't stripped)

[jfbastien]

My main concern is still that wasm would be different from other LLVM toolchains.

There's also quite a bit of ongoing work in the LLVM community around LTO, so if we want to change things we should probably get involved ~ now :-)

[lukewagner]

Question/Idea: definitely on native, "-shared"/"-fpic", negatively impact codegen, so you wouldn't want to pass them unless necessary. However, for WebAssembly, I can't think of any examples where that would be the case. So what if, for WebAssembly, normal (non "-shared") compilation was toolchain specific and "-shared" produced a standard library format that could then either be statically or dynamically linked. Thus, for WebAssembly, "-shared" meant "sharing" not "dynamically linked" and the consumer of a -shared library could choose what to do.

[kripken]

@lukewagner, are you saying we would go wasm => LLVM IR, then do LTO on that? This seems a very interesting idea! I worry @jfbastien wouldn't like it either though, because it is a nonstandard way to do things? Putting aside the non-standardness, I like it in principle, but am worried about the details. Namely,

• LTO, already a slow thing, would be slower. We would need to translate wasm to LLVM IR, and in particular go from a register-based AST to SSA form basic blocks, and so forth.
• It would be "fun" to translate debug info in this reverse direction. (Maybe debug info isn't generally used on LTO builds, but once someone has an LTO-only bug, they might really want that debug info.)
• I am sure you are right in that we would need to add an LTO section with some new stuff. It isn't obvious to me what would go there (aside from obvious stuff like optsize and minsize annotations), this seems like a deep topic to investigate, and worse, will likely be an ongoing thing, as LLVM improves. For example, if LLVM adds some new form of metadata that helps LTO, we would need to add it as well.

Overall, I love the elegance of the idea. However, those three concerns worry me.

@jfbastien, definitely, if there are LTO changes happening in LLVM now, it seems like we should get involved. Are there discussions on the mailing list?

[lukewagner]

@kripken Yes, in my second-to-last comment that is exactly what I was suggesting. In my last comment I was then wondering if "-shared" could be the "give me the standard, interoperable output" flag that could be used for both static linking (w/ LTO if the wasm=>LLVM IR idea works) and dynamic linking. That way, non-"-static" would be free to do whatever was natural for the toolchain.

[kripken]

I agree with most of the last of those two comments, yes, I see no reason why -shared could not be used to generate libraries for either static or dynamic linking, in some agreed-upon format.

This still leaves open what that agreed-upon format should be, and what the object format should be - as it must contain what is needed for the the -shared format. Which leaves the open questions about the 3 proposals (AST serialization, fat objects, wasm2llvm).

Using -shared for both static and dynamic linking doesn't mean the two libraries are the same, though - probably you didn't mean that, but just to be sure. It seems natural for us to do what native build systems do, namely support static libraries using .a containers, and dynamic libraries without that container.

[lukewagner]

What I was thinking is that, to do the native thing (use .a containers), you don't pass "-shared". Thus, the only really "new" thing would be to allow dynamic .wasm libraries to be passed in as static link libraries (as an alternative to .a).

As for the format, since this is the same format that would be decoded natively by the browser in the case of dynamic linking, for obvious practical reasons we'd want it to be basically the same module structure as normal non-dynamically-linked modules (there would be some differences around heap-init, e.g., but I wouldn't expect too much else).

[kripken]

I agree on the individual points: I think there is no reason to disallow linking a dynamic library statically; and dynamic libraries should be similar to non-dynamic wasm "executables", since both are run by the browser (but as you say, might be some heap init differences, due to one being shared). But I don't think I see your overall point, or maybe we haven't spelled out the details yet. Here is how I see things:

• There is an object file format, something that source C and C++ files compile into. You can link some of these to get another of them that contains them all, and we want that linking to be fast, like native build systems.
• There is an executable format. An object file(s) can be transformed into this. This is what the browser can run. We want the conversion of an object file to an executable to be fast, like native build systems (incremental compilation, etc.). We also want it to be super-easy to perform LTO, as the executable is what is sent over the wire, and we are working hard to reduce that size in multiple ways, of which LTO is one.
• There is a static library format, a .a container. It bundles object files. Linking is per the normal native build system .a semantics (pull out just the needed objects and no more), so it is fast and efficient (more efficient than just linking those files to one big object file, in general).
• There is a dynamic library format. It can be created from a bunch of object files, with the -shared flag. When we have dynamic linking, this will be runnable by the browser, and in general will be quite similar to the executable format, in that we want generating it to be fast like native build systems, but also super-easy to LTO as well.

The executable and dynamic library formats are things that would be standardized in the WebAssembly spec - they are things that the browser will run. They would be "wasm" files that websites contain, like they contain "html" and "css" files. On the other hand, the object and static library formats are something we would like to unify as much as possible, to avoid fragmentation, but in principle there might be multiple toolchains with different design decisions in this space - e.g. one focused on incremental compilation speed at all costs, or one focused on LTO efficiency at all costs.

I think the fat objects proposal from before is a good way to achieve the goals, given this perspective on things (it would avoid fragmentation in that it provides near-optimal results on both incremental compilation speed, and LTO efficiency; and is easy to implement). But perhaps we have a different view and/or goals, though?

[BrendanEich]

Hi -- long-time fan, first-time caller here -- I love you all! Hope this is helpful.

I survived the RPC wars of the mid-late-80s, between (mainly) Sun Microsystems and Apollo Computers (DEC spinoff). Sun advanced a data serialization standard called XDR, which was canonical and simple. Apollo advanced a complex, "receiver makes it right" ensemble of target-server-specific data formats, to optimize f.p. in the pre-IEEE754 days. Believe it or not. I still remember Tom Lyon of Sun arguing before an IETF meeting that Sun XDR was better, because you could put it on a shelf for 20 years, and reliably (assuming no zombie/robot apocalypse) deserialize it. Apollo's target-server-dependent scheme made that less likely to be reliable, or available at any rate.

It seems to me the same argument applies here. If we want LTO and wire it directly to LLVM, we'll have to be willing, in 20 years, to do without it and suffer whatever perf hits that entails. If we wire in any deep way to today's LLVM, the odds go up of unintended dependencies that break in 20 years.

Sorry to harp on 20 years, but JS is 20 years old this month. While CSS table layout changes (vs. spacer GIFs) may have broken content from back then, and web.archive.org doesn't go back quite that far, now we know better than to think that WebAsm won't last 20 years. From watching businesses fail to maintain their sites, I would bet real money that there'll be real WebAsm files around then that are 18+ years old.

Signed, your (all of you) #1 fan!

/be

[kripken]

I see what you mean. But I would say that LTO is different from the general "tech to be supported in 20 years" in two respects: (1) it does not affect semantics / it is "optional" in that content will run fine without it, (2) we really want it at its maximal efficiency as much as possible.

With fat objects, an object file will always be usable, as long as wasm exists and the sun still rises. We never lose that.

With a non-LLVM LTO approach, we lose the maximal efficiency of LTO, in return for that weaker LTO working forever. That puts us in an odd middle ground: we can do some amount of LTO on it, and we can do that in the very long term; but we've given up quite a lot to get that. Instead of using the full power of LLVM IR and LLVM's LTO - which are constantly evolving - we standardize something fixed and can use only that.

But the issue here is really one of fragmentation. Let's say that we decide to standardize some weaker form of LTO in our object format. If it isn't as good as the full power of LLVM LTO, which will either be true immediately or become true eventually, then some people will use a toolchain that does have the option to use all of LLVM LTO. The final output of that toolchain will still be wasm executables, so everything is ok, the web is safe. The only loss is that there are multiple toolchains for developers. All I am proposing with fat objects is a way to avoid that toolchain fragmentation.

[BrendanEich]

I like Luke's square-the-circle non-compromise (but for recovering LLVM IR from .wasm) compromise, don't get me wrong. Would want MS folks to bless it, of course.

/be

[lukewagner]

@kripken In the context of the list you gave, I'd describe my proposal as allowing the dynamic library format to be used as an alternative to .a/.lib as a (portable, standardized) static library format.

I still don't see any real reasons why a .wasm file couldn't support good LTO in LLVM by translation to LLVM IR. From your list of concerns: I think the translation time will not be remotely significant compared to the rest of LTO (Amdahl's law); bebug info would be fun, but I expect we wouldn't have to solve this immediately b/c debugging usually wants incremental builds anyway; the last point wasn't really a concern. To make this conversation more concrete, it'd be nice to identify specific semantic information which LTO measurably benefits from and that WebAssembly is lacking.

[kripken]

Dynamic libraries and in general just linking together object files are an alternative to .a files, of course. But they are not as efficient in many cases. Example: a .a that contains a .o with a global initializer function. If .a linking avoids bringing in that .o file, we are safe from it; but if we already linked all the .o files, it will be there, and we depend on LTO being able to extricate it. LLVM LTO is quite good, but currently fails to do so. Could it be fixed? With enough work, sure, just like all the other things; maybe some day .a files will be obsolete, and native build systems can remove them.

Regarding translation time: While full LTO, doing most of -O2 style optimizations, would be similar to what you suggest (LTO time dwarfs all the rest), there is also a faster LTO which focuses on code size, and just internalizes everything, and runs globalopt and a few other select passes (this is what Emscripten does). Those are O(module), the same complexity as wasm => LLVM IR would be. Hence my concern.

Examples of specific semantic information for LTO would be:

• optsize and minsize, and gradients of those.
• More fine-detailed metadata that LLVM can provide could be which basic blocks are expected to rarely run (based on expect and other intrinsics).
• PGO data is another topic on its own, this was still being worked on last I heard (perhaps related to what @jfbastien was saying about current LTO work?).
• There is also the matter of LLVM intrinsics being "lost": In IR, we know that @llvm.some.instrinsic is a magic function doing something, but we might lower it into a bunch of instructions in wasm; to go back, we'd need to pattern match somehow (or mark them as a group somehow).
• Then there are of course the linkage modes, like likeonce_odr, but I assume the object format would have them anyhow (but not the executable format).

Of course, one simple solution to all this is to just allow our object files to have some opaque JSON metadata, and then we'd just translate LLVM IR to that metadata when generating wasm, and translate it back to LLVM IR to perform LTO. My proposal of fat objects is identical to this, except that we replace JSON with LLVM IR, and we avoid the need to translate to and back from it.

[lukewagner]

First, on translation time: even in the O(module) case, I don't see how the trivial process of building in-memory LLVM IR from wasm would be comparable to the rest of the pipeline. I mean, LLVM bitcode doesn't just mmap into memory, it has to be decoded as well (and wasm is a smaller format than .bc iiuc) so I have a difficult time believing this would be a significant.

All the other points are interesting.

• For the global initializer case, this sounds like a feature that actually makes sense for dynamic link libraries: partition functions and global initializers so that the global initializers are only run when one of their partition's functions is imported.
• optsize, minsize, basic block coldness all seem like pretty straightforward and small additions to an LTO section and also fairly general
• the intrinsic point is a good question and seems related to Clarify intrinsification criteria #29: if there are non-obvious higher-level semantics associated with the intrinsic, we might want to consider having that be an intrinsic in WebAssembly (for the dumb-compiler-which-spits-out-poorly-optimized-code use case), or at least the hot ones.

[jfbastien]

Stepping back a bit: do we need object files and shared libraries to be wasm at all?

I'm really strongly advocating for whatever we do to not be different from what existing compilers do. If we diverge, we should do so while advancing the state of compilers for other targets.

We will need to figure something out for our toolchain, but do we need to do so now?

Re: LTO and PGO work, the GCC folks are working on bringing their thinLTO work to LLVM (see EuroLLVM 2015 and recent llvm-dev discussions). They're also independently working on things such as AutoFDO and eventually PGO. ThinLTO in particular stands to change what LLVM does for LTO by duplicating likely-hot call trees across object files, but otherwise doing parallel compilation that's fully cache-able.

On debug info and profile info: I'd drop it from wasm entirely until we figure out how to make metadata relevant, stable, and standardizable. This will be hard. FWIW DWARF is Turing-complete.

[kripken]

I think having object files contain wasm AST code is in the spirit of "do what existing compilers do". x86 object files contain x86 code, wasm object files should contain wasm code. This seems the best possible way to get incremental compilation, in particular.

(The fat objects proposal just states that we also keep around the LLVM IR, so it is available for LTO. This in fact fits in with how LLVM does LTO now, which is be able to link files containing both native code and/or LLVM IR - fat objects are nothing more than objects that contain both, by default, for the wasm case.)

Note btw that I am not saying that object files need to be wasm executables, with all the macro compression and other aspects of "something shipping for the web", of course - perhaps there is an ambiguity here. That's why I wrote out a difference between wasm executables and wasm object files above. Objects might also contain a bunch of stuff we don't want in final executables, like metadata.

I agree we don't need to figure out something for the toolchain now. We can also avoid figuring it out later, in fact - as I said, maybe we will just have multiple toolchains, if we can't all agree on one. A goal is to enable multiple compilers, after all.

[lukewagner]

Agreed that .o/.a files don't need to be proper .wasm, that's just a tangent I went off on.

@kripken Yes, we do want to have multiple toolchains, but I think it would be a serious lost opportunity if libraries (static or dynamic) couldn't be shared between them.

[lukewagner]

In the light of #74, though, I wonder if the standard for static link libraries could just be a subset of ELF and the .a format. I don't know the details of either, though, and I'd be curious what Microsoft people think.

[kripken]

Side note, over lunch @sunfishcode convinced me that compilation time is not an issue for the wasm => LLVM IR proposal, as wasm is designed for fast parsing, and basic dce done on it before even emitting LLVM IR for LTO might or might not end up being faster.

[MikeHolman]

I think a decision here should be delayed as much as possible. If we want a standard format, then we should wait until people from multiple toolchains have a chance to weigh in.

From our side, MSVC will take some time before they decide whether they want to support wasm, and we don't want to cause fragmentation by deciding on something too soon. But more generally, if we want multiple compiler backends emitting wasm, we should avoid making any decisions which might make it uncomfortable for them, or make them feel that wasm is first and foremost an LLVM target.

I don't know who else would be interested in a wasm backend, but at the very least I think we should hold off on any decision until we go public and give people a chance to weigh in.

[lukewagner]

I agree. My goal with creating this issue was just to discuss the question of should we have a standardized ABI and, if so, when should we standardize it.

When we add dynamic linking, the ABI issue will be a lot more prominent than v.1, so currently I'm thinking it's probably fine to wait until then, at which point in time we'll also have a lot more experience.

[sunfishcode]

That sounds good to me too. @MikeHolman, would you be willing to draft a few sentences about this that we could include in V1.md?

[lukewagner]

I forgot I left this open; I think it's resolved.