Summary 📘

This PR adds support for unaligned bit arrays on the JavaScript target.

In expressions:

• Arbitrary sized integer segments:

  <<1:4>>
  <<12:9-little>>
  <<12:29-big>>
  <<1234:100-little>>

• Arbitrary sized bits segments:

  <<<<0xABCD:15>>:bits-10>>

In patterns:

• Arbitrary sized int segments:

  let assert <<_:7, i:19-little-signed>> = <<0xABCDEF12:26>>

• Sized and unsized bits segments:

  let assert <<_:7, a:bits-3, b:size(14)-bits, c:bits>> = <<0xABCDEF:24, 0x1234:16>>

There is a compiler warning if the above features are used when gleam.toml specifies a version < v1.9.0.

Implementation Details 🛠️

• The BitArray class in the prelude now has bitSize, byteSize, bitOffset, and rawBuffer properties.
  • The value of any unused high bits in the buffer's first byte, and any unused low bits in the buffer's final byte, are undefined.
  • Public API for use by FFI code is now: bitSize, byteSize, bitOffset, rawBuffer, byteAt(index).
  • Deprecated APIs still used by existing FFI code: get buffer(), get length(). Using these emits a deprecation warning at runtime, and throws an exception if the bit array is unaligned (i.e. bitOffset isn't zero or bitSize isn't a multiple of 8).
• JSDoc annotations have been added to allow type-checking by adding // @ts-check to the top of the file.

Implications for @external JavaScript code 🌍

• All existing JavaScript FFI code that operates on bit arrays needs to be updated to support unaligned bit arrays.
• Code that isn't updated will:
  • Continue to work, provided it is only used with aligned bit arrays.
  • Emit deprecation warnings at runtime on use of BitArray.length and BitArray.buffer.
  • Throw a runtime exception if these deprecated APIs are called on a unaligned bit array.
• No existing code breaks because unaligned bit arrays on JavaScript weren't previously possible.

Implications for gleam/stdlib 🤝

• I have the updates for gleam/stdlib ready, mostly affecting gleam/bit_array. It can only be merged once this PR goes in as its tests won't run on stable Gleam. It may be necessary to run the new stdlib tests on nightly for a short period, with them segregated into their own file so they can be included/excluded depending on the active Gleam version. I'll sort that out once this PR makes it through review.
• Future stdlib versions that support unaligned bit arrays on JavaScript work fine on earlier Gleam versions, there are no compatibility concerns there.

Testing 🧪

There's certainly some complexity and tricky bitwise operations here, mostly in the JavaScript prelude. The following has been done to ensure correctness:

• Many new tests added to language_tests.gleam, and test/javascript_prelude.
• Every path and branch through the code that performs slicing, concatenation, and conversion to ints/floats is covered by at least one test.
• Extensive fuzzing has been performed on these operations which validated millions of combinations of bit array contents, segment sizes, offsets, endianness, signedness, etc. on JavaScript against the result on the Erlang target.
• Issues found during fuzz testing were fixed and added to the language tests and prelude tests.