Type-safe C FFI

[yorickpeterse]

Summary

The current FFI is not developer friendly, and inefficient. We should replace it
with an API wherein developers declare the foreign types/functions in an
extern block. This block is then turned into a module, which in turns contains
methods/objects/etc for the C types. This allows for a type-safe and more
efficient FFI.

Motivation

The FFI API we currently provide with std::ffi is rather clunky, and more of a
proof of concept than a production-ready API. For example, accessing the exp()
function from libm is done as follows:

import std::ffi::Library
import std::ffi::types

let libm = Library.new(Array.new('libm.so.6', 'libm.so'))
let exp = libm
  .function(name: 'exp', arguments: Array.new(types.f64), returns: types.f64)

exp.call(25.1234) as Float

This API is verbose, inefficient, and not type-safe.

It's verbose because of how argument types are specified, and the type-casting
necessary when calling a function. It's inefficient because function call
arguments are first packed into an array, which the VM FFI code then copies into
a libffi arguments list. When converting types we also run code that is built to
support all possible cases of input types, even when the input type is always
the same (e.g. a Float above).

It's not type-safe because everything happens at runtime. This means the
following is valid at compile-time:

exp.call('blorb')

I propose that we introduce an extern keyword. This keyword can be used to
declare a set of foreign types and functions. These symbols are then
type-checked by the compiler, allowing for:

1. Type-safe access to C data
2. Better code generation for function calls

The above example would translate to the following:

let libm = extern('libm.so.6', 'libm.so') {
  def exp(argument: Float) -> Float
}

libm.exp(25.1234)

The argument of extern is a list of library names to load. The body is used to
define the functions (using the required method syntax), structures, constants,
etc.

If extern reduces a certain type of which the compiler knows its contents
(e.g. the methods), it can then ensure type-safety when using the library. In
addition, because the method signatures are known at compile-time we can
generate better code for calling C functions.

Implementation

extern is a special keyword, and its body allows a restricted subset of the
Inko syntax. Its return value is an anonymous module. Functions defined in the
extern body are exposed as module methods, while structures are exposed as
objects as if defined using the object keyword. Constants can be defined using
let, but their values are limited to

For enums we don't have any syntax, as Inko doesn't have enums. For this we
probably would require the use of regular constants. For example:

enum Letter {
    A, B, C
}

Translates to:

extern {
  let A = 0
  let B = 1
  let C = 2
}

The extern block is just an expression, and its return value is a module. The
compiler is aware of what constants/methods/etc the module provides, due to the
specialised syntax used in an extern block. Methods can be called like any
other. For constants we need to reintroduce the foo::Bar syntax for accessing
constants. So for the above case:

let lib = extern {
  let A = 0
  let B = 1
  let C = 2
}

lib::A # => 0
lib::B # => 1

For pointers we'd have to import std::ffi::Pointer:

import std::ffi::Pointer

let libc = extern('libc.so.6') {
  def malloc(size: Integer) -> Pointer
  def free(pointer: Pointer)
}

libc.free(libc.malloc(32))

A challenge is handling the differences in platforms. At compile-time we don't
know if the code is going to be running on Windows, Linux, both, or something
else. This means we can't use compile-time directives/pragmas that say "load
this only on Windows", instead that needs to be deferred to the run time. But
for the compiler to guarantee type-safety, it needs to know for every library
what types/methods/etc there are. In practise this means we need a separate
extern block for every platform difference. For example:

let LIB_WINDOWS = extern('libc.dll') {
  def foo(a: Integer, b: Integer) -> Integer
}

let LIB_UNIX = extern('libc.so.6') {
  def foo(a: Integer) -> Integer
}

This wouldn't work though: if extern loads the library upon definition, the
LIB_WINDOWS line would fail on Linux as libc.dll doesn't exist there. We could
do something like this:

import std::os

let LIB_WINDOWS = extern('libc.dll') when os.windows? {
  def foo(a: Integer, b: Integer) -> Integer
}

let LIB_UNIX = extern('libc.so.6') when os.linux? {
  def foo(a: Integer) -> Integer
}

Here the extern would only load the library if the when ... condition
evaluates to true. If it's not true, the return value would be some sort of stub
library. Using functions of a stub library produces a panic. But what about
constants? We can't run code upon accessing constants, unless we force
developers to using a get method of sorts. So instead of this:

let libc = extern('libc.so.6') {
  let A = 10
}

libc::A

You'd have to write:

let libc = extern('libc.so.6') {
  let A = 10
}

libc.get('A')

The benefit is that we can tap into get to panic for a stub, and that we don't
need the libc::A syntax. The downside is that we lose type-safety, as get
doesn't know its return value may differ based on the constant name it's given.

An option is to not define any methods/constants/etc on a stub, and accessing
anything from the stub just triggers a regular panic triggered when accessing a
non-existing method/attribute.

For function calls ideally we find a way to generate more optimal code. So if a
function only takes two floats, we should generate code such that we don't check
for all the other possible argument types. We also shouldn't need an
intermediate arguments Array. How exactly we'd achieve this I'm not sure yet.

We'll need to spend more time thinking about how we approach these problems
before we implement anything of this proposal.

Drawbacks

We have to make changes to the syntax, and probably add some additional VM
instructions. We also likely need to rework the FFI internals a fair bit.

Tasks

• Add support for conditional compilation at the import level, supporting the architecture and OS initially (ABI is less important for the time being)
• Figure out what syntax to use for defining C functions, types, and the names of the libraries to link against
  • Note that C libraries may use names that aren't valid Inko identifiers, so if we use import to specify the C library names, it must use single/double quotes for the name
• Add parser support
• Add HIR support
• Add type checking support
  • Disallow passing C types to anything that isn't a C type itself (e.g. we can't pass them to generic type parameters)
• Add MIR support
  • Split GetField and SetField into GetField/SetField and GetStructField/SetStructField
• Add LLVM support
• Collect the C libraries to link against and pass them to the linker

[yorickpeterse]

What we could do is this:

Extern types/functions are defined using this syntax:

extern 'libsqlite3.so', 'sqlite3.so', 'libsqlite3.dll' {
  # A constant/global
  let SQLITE3_OPEN_READONLY: Int
  let SQLITE3_OPEN_READWRITE: Int
  let SQLITE3_OPEN_CREATE: Int

  # A C structure
  class Sqlite3 {
    let @foo: A
    let @bar: B
  }

  # A function
  fn sqlite3_open_v2(path: String, handle: Pointer, flags: Int, vfs: Pointer) -> Pointer
  fn sqlite3_close_v2(handle: Pointer)
}

The symbols are then exposed to the module like any other. A class in an extern can only define fields, and an fn can't define a body.

There are no #[cfg(...)] tags or anything like that. Instead, import supports a platform condition. This means conditional compilation is handled at the module level, not the symbol level. This is easier to type-check for, and promotes/forces pushing platform specific code into its own module. I'm not yet sure what the import syntax would look like, but I'm thinking of one of these:

import if windows std::stdio::STDOUT

# or:
import std::stdio::STDOUT if windows

[yorickpeterse]

#508 removes the old libffi FFI, meaning this is becoming more important. I'm currently leaning towards handling platform differences at the module level as outlined in #290 (comment). A full DSL for conditional compilation gets out of hand fast, and I like the idea of forcing platform specific code into dedicated modules (a pattern already common in Rust itself).

[yorickpeterse]

We won't support compiling C files and linking against the result. Instead, we'll only support linking against existing libraries. I want users of Inko to avoid using C as much as possible, and I don't want to develop yet another language specific build tool.

[yorickpeterse]

For imports, I'm thinking of the following:

• import "foo" dynamically links against libfoo.so
• import static "foo" statically links against libfoo.a

One challenge here is that some platforms don't ship both static and dynamic libraries of the same project. For example, some distributions only ship dynamic libraries for LLVM, while others ship both static and dynamic ones. This means that if you want static linking, some sort of fallback is necessary.

In Rust and possibly other languages, this is handled using complicated build scripts, but I don't want such a setup for Inko.

[yorickpeterse]

Another option is that we default to static linking for everything, and allow changing this to dynamic linking for everything using a flag (e.g. --dynamic). This means there wouldn't be an import static "foo", only import "foo".

The benefit here is simpler syntax, and developers packaging Inko programs can still decide to dynamically link everything if desired. The downside is that this would apply to all C libraries in a program, not just a specific one.

A variation of that is to support --dynamic=LIB, allowing you to enable dynamic linking for a specific library, and --dynamic=all to enable for all libraries. This way you can e.g. static link everything except for LLVM, which you would dynamically link. The problem here, and with mixed linking in general, is that I don't know how well this works in practise.

Another approach is to invert this: we dynamically link against C libraries by default, as dynamic libraries are likely more commonly available compared to static libraries. If static linking is desired, one could use inko build --static to statically link all C libraries included. This is probably a better default compared to static linking.

[yorickpeterse]

On GNU systems we can use -l:libX.a to favour a static library over its dynamic counterpart: https://stackoverflow.com/questions/6578484/telling-gcc-directly-to-link-a-library-statically. For macOS I'm not sure what the equivalent is, if any.

[yorickpeterse]

For the compiler there's a challenge/obstacle: C pointers are recursive (e.g. *mut *mut u8 using Rust syntax), which means whatever type we use to express C types has to be recursive. Usually you'd do e.g. Pointer(Box<Ctype>) where Pointer is a variant in the Ctype enum, but we can't use this because TypeRef is Copy, and Box<T> isn't Copy.

So we need a way that allows TypeRef to support recursive C types, while still keeping it Copy (something we rely on literally everywhere).

[yorickpeterse]

I thought about and briefly experimented with using tagged integers to handle this. My thought was that by using a 64 bits integer, we can use 32 bits for the payloads/IDs (which the compiler already uses for most of its IDs), and the extra 32 bits for tags.

But this too sadly doesn't support infinitely recursive pointer-pointers, as you'd need an extra bit for every pointer in the pointer-pointer chain. Of course such types are rare, though technically not unheard of.

We could take the approach of just not supporting pointers more than two levels deep (so Pointer[Pointer[T]] is the maximum), but I'm not sure what practical problems that may cause in the future.

[yorickpeterse]

I'm currently leaning towards just not supporting more than two levels deep. This way we can just use a regular enum, and still support 99% of the C code out there.

[yorickpeterse]

Actually that doesn't entirely solve it either: the data structure still needs to be recursive to support e.g. Pointer[Int]). Adding variants for all those base types (e.g. IntPointer(size) and FloatPointer(size)) gets messy.

[yorickpeterse]

We can combine the two: we use integer tagging, and limit ourselves to at most pointer-pointers. In this case we can use the following bit patterns (where xxxx is the upper 32 bits):

xxxx 0000 = IntN, N in the upper bits
xxxx 0001 = FloatN, N in the upper bits
xxxx 0010 = StructN, N is the struct ID in the upper bits
xxxx 0100 = Pointer[IntN], N in the upper bits
xxxx 0101 = Pointer[FloatN], N in the upper bits
xxxx 0110 = Pointer[StructN], N in the upper bits
xxxx 11xx = Pointer[Pointer[X]], based on the first two bits

Since payloads are limited to 32 bits, we have up to 32 bits we can use for tagging. This should be more than enough if at some point we want to support e.g. Pointer[Pointer[Pointer[Int]]].

Fiddling with the bits is a bit annoying, but it allows the type to remain Copy, which is needed if we want to stick it in TypeRef.

[yorickpeterse]

For dereferencing I'm thinking of using this syntax:

ptr.0      # Read the pointer
ptr.0 = 42 # Write to the pointer

This way we don't need to introduce extra syntax. C doesn't allow numbers as field names if I'm not mistaken, so this shouldn't cause any conflicts. At some point we could also extend this to support arbitrary offsets (e.g. ptr.41 = bla), but we won't bother initially.

For pointers to structs you'd just use the usual pointer.field_name syntax for both reads and writes.

[yorickpeterse]

Currently I'm playing around with this setup:

#[derive(Debug, PartialEq, Eq, Clone, Copy, Hash)]
pub struct Ctype {
    pub kind: CtypeKind,
    pub value: u32,
}

#[derive(Debug, PartialEq, Eq, Clone, Copy, Hash)]
pub enum CtypeKind {
    Int,
    Float,
    Struct,
    Pointer(PointerKind),
    PointerPointer(PointerKind),
}

#[derive(Debug, PartialEq, Eq, Clone, Copy, Hash)]
pub enum PointerKind {
    Int,
    Float,
    Struct,
}

#[derive(Debug, PartialEq, Eq, Clone, Copy, Hash)]
pub struct StructId(pub u32);

This works and doesn't need any bit tagging, but it's a bit messy. Technically PointerKind could be folded into CtypeKind, but then we'd end up with variants like IntPointerPointer, FloatPointer, StructPointerPointer, etc, which isn't necessarily better.

Not making TypeRef a Copy tag won't work either due to the sheer number of compiler errors this introduces, and TypePlaceholder requiring it to be Copy for Cell.get to be available.

[yorickpeterse]

For type conversion, we'll go with the following:

C types aren't implicitly converted to Inko types, instead you have to use methods such as Int.from_c and Float.from_c. Initially these will use tagging for Int and heap allocations for Float, but after #525 they would be turned into bitcasts. We'd support these methods:

• String.from_c(ptr) -> Option[String]: returns an Inko String from a Pointer[Int8], if the input is valid UTF-8.
• Int.from_c(val) -> Int turns an Int8/Int16/etc into an Inko Int
• Float.from_c(val) -> Float same but for Float
• ByteArray.from_c(ptr, len) returns a ByteArray from some pointer, not sure yet about this

For Int.from_c and Float.from_c to work, we'll need to allow upcasting e.g. Int8 to Int64 implicitly, so we don't need Int.from_int8, Int.from_int16, etc. As that's always possible (e.g. Int8 to Int32 never loses information), we can just do this as a bitcast.

An alternative is that instead of Int.from_c and Float.from_c, we allow x as y where x and y are C types, e.g. some_value as Int32. This however requires that the MIR for type casts generates code to box these values, for now we're still using tagged integers and boxed floats.

The benefit of explicit casts is that in the compiler there's only one place where we need to cast these types, and that's in the intrinsics used for this; instead of having to check for this in a whole bunch of places.

[yorickpeterse]

For the foreign type names, we'll have the following:

• Int8, Int16, Int32, Int64, Size
• Float32, Float64
• Void, Pointer[T] and Pointer[Pointer[T]]

I'm not sure yet how we'll expose these names to type signatures. I don't want to always make them available as that clutters the namespace. I also don't like the idea of using a pseudo parent constant/module, e.g. c::Int8 or C::Int8, as that results in very tedious-to-write code.

[yorickpeterse]

Come to think of it, the pseudo package approach seems best after all. It would work as follows:

• To access C types, you have to add import c to your module
• You refer to C types using e.g. c::Int8 or c::Pointer[c::Void]. This is verbose, but that should signal one should be avoiding C code in the first place
• You can't import the types as they aren't actually defined, rather it's just a trick so we don't need special syntax for C types, and can still detect them when looking up symbol names
• The c module is defined in the compiler with a known ID (i.e. ID 0) so we can easily check for it as part of symbol resolution