Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
refactor(tokenizer): Iterate over char codes (#894)
This leads to a ~43% speed up in tokenizing performance 🤯 ``` Old Tokenizer x 29,499 ops/sec ±0.20% (93 runs sampled) Tokenizer x 42,260 ops/sec ±3.00% (92 runs sampled) ```
- Loading branch information