Skip to content

Python v0.2.0
Compare
Choose a tag to compare
@n1t0 n1t0 released this 20 Jan 14:24
python-v0.2.0
da7e629

In this release, we fixed some inconsistencies between the BPETokenizer and the original python version of this tokenizer. If you created your own vocabulary using this Tokenizer, you will need to either train a new one, or use a modified version, where you set the PreTokenizer back to Whitespace (instead of WhitespaceSplit).

Assets 2
Loading