Add more support for tiktoken based tokenizers#1493

Merged

ArthurZucker merged 19 commits intomainfrom update-bytelevel

Apr 15, 2024

+134-8

Commits on Apr 15, 2024

first commit
ArthurZucker
committed
update
ArthurZucker
committed
clippy
ArthurZucker
committed
lint
ArthurZucker
committed
clippy and lint
ArthurZucker
committed
fmt
ArthurZucker
committed
revert print
ArthurZucker
committed
😈
ArthurZucker
committed
style
ArthurZucker
committed
add a test
ArthurZucker
committed
more fmt
ArthurZucker
committed
Use ignore_merges
ArthurZucker
committed
stub
ArthurZucker
committed
fix
ArthurZucker
committed
update
ArthurZucker
committed
Update tokenizers/src/models/bpe/model.rs

ArthurZucker
and
Narsil
authored
update
ArthurZucker
committed
rust lint
ArthurZucker
committed
dob; t repeat yourself
ArthurZucker
committed