You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There might be some changes which might make it better, like using None instead of "no_regexp" in Python.
(In RUST, since the regexp is hardcoded, and we definitely don't want to enable users to choose, since it should be done as another pre_tokenizer if you want custom regexp, going with the simple Enum instead of Option seems more correct as a first approach.
The text was updated successfully, but these errors were encountered:
This is sort of done here:
https://github.com/huggingface/tokenizers/pull/896/files#diff-c2be5839194fbc7e052d4600684eeb35f0db504b5b4df6565f01057762e23774R46
There might be some changes which might make it better, like using
None
instead of"no_regexp"
in Python.(In RUST, since the regexp is hardcoded, and we definitely don't want to enable users to choose, since it should be done as another pre_tokenizer if you want custom regexp, going with the simple
Enum
instead ofOption
seems more correct as a first approach.The text was updated successfully, but these errors were encountered: