dart:ffi int8 etc truncate and sign extend vs exceptions

[dcharkes]

What do we do with Dart ints which do not fit in the C++ type they are stored into.
And do we want the behavior to be consistent between Pointer.store() and Pointer.asFunction()?

Throw exception

Pointer<Int8> p = allocate();
p.store(128); // throws exception

typedef NativeFunc = ffi.Int64 Function(ffi.Int8);
typedef DartFunc = int Function(int);

DartFunc f = somePointer.asFunction<DartFunc>(); // simply returns its input
f(128); // throws exception before running C function

typedef NativeFunc2 = ffi.Int8 Function(ffi.Int8);

DartFunc f2 = somePointer.asFunction<DartFunc2>(); // multiplies its input by 2
f(127); // throws exception after running C function, on return value

pro: safe
con: might be slower
con: Inconsistent with our handling of uint64, as there we interpret the bits as int64 instead of throwing exception. #35757

Truncate and sign extend

Pointer<Int8> p = allocate();
p.store(128); // truncated when written into C memory
Expect.equals(-128, p.load<int>()); // sign extended when read back into Dart

DartFunc f = somePointer.asFunction<DartFunc>();
Expect.equals(-128, f(128)); // 

Trampoline truncates and sign extends argument passing -128 as 32/64 bits to C based on architecture

DartFunc f2 = somePointer.asFunction<DartFunc2>(); // multiplies its input by 2
Expect.equals(-2, f(127));

Various (or all) calling conventions do not truncate and sign extend values smaller than 32 bits, so it returns 254 (0xFE) which is sign extended by the trampoline as -2.

pro: fast
con: unexpected behavior

[lrhn]

We want values that do fit in a smaller type to be round-tripped safely, so extracting a value from a signed type must sign-extend. If you store -2 in an Int8, the bits stored will be 0xFE. Reading that back as -2 means sign extending.
There is no way:

f(127); // throws exception after running C function, on return value

can work. The function is given a valid Int8 value, and it returns a valid Int8 value. You need to convert that valid Int8 value to an int. There is no way that you can see the overflow that happened on the other side. The returned result is not 254, the result is the bit pattern 0xFE interpreted as Int8, aka -2. You can convert that to int.

So, the only real question is what to do when storing something that doesn't fit.
The dart:typed_data lists truncate on store (and sign extend when extracting from a signed type, zero extends from an unsigned type). That is also the behavior you get in C code when storing an int in an int8_t, so its unlikely to be surprising.

It seems to be the least suprising choice. If the integer value fits in the size and signedness, truncating and sign-extending will give the original value back (and for unsigned types, truncating and zero-extending will also give the original value back, if it fit in the smaller unsigned type to begin with).

[dcharkes]

We will go for truncating and sign extending everywhere.

[dcharkes]

Implementation is tracked in #36122