AddedToken
's argument are ignored when called in add_tokens
's method of slow tokenizers
#20734
Closed
2 of 4 tasks
Labels
Core: Tokenization
Internals of the library; Tokenization.
System Info
transformers
version: 4.25.1Who can help?
@ArthurZucker
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
The explanations of the bug and its reproduction are contained in the following google colab: https://colab.research.google.com/drive/19SS6Tzlgo0vntFtM6ZsCYq8BNZ5Dy1cS?usp=sharing
Expected behavior
I would expect the fast and slow tokenizers to treat the
AddedToken
's arguments in the same way.I think the loss of information for the slow tokenizer occurs at this line:
transformers/src/transformers/tokenization_utils.py
Line 411 in a413c72
The text was updated successfully, but these errors were encountered: