Token classification pipeline results different with tokenizers==0.11.6 vs tokenizers==0.12.0

[davidmezzetti]

I'm not sure if this is an issue with transformers, an issue with tokenizers or expected behavior. But when running the token classification pipeline with an aggregation_strategy="simple" the results are slightly different with tokenizers==0.12.0.

The following code produces different results (both examples use transformers==4.17.0).

from transformers import pipeline
nlp = pipeline("token-classification")
nlp("Hugging Face Inc. is a company based in New York City", aggregation_strategy="simple")

With tokenizers==0.11.6:

[{'entity_group': 'ORG', 'score': 0.99305606, 'word': 'Hugging Face Inc', 'start': 0, 'end': 16}, {'entity_group': 'LOC', 'score': 0.9988098, 'word': 'New York City', 'start': 40, 'end': 53}]

With tokenizers==0.12.0:

[{'entity_group': 'ORG', 'score': 0.99305606, 'word': ['Hu', 'gging', ' Face', ' Inc'], 'start': 0, 'end': 16}, {'entity_group': 'LOC', 'score': 0.9988098, 'word': ['New', ' York', ' City'], 'start': 40, 'end': 53}]

[patil-suraj]

cc @SaulLu @Narsil

[Narsil]

There's a proposed PR here : #16537

[Narsil]

Tokenization tests were run before releasing 0.12 but not the pipeline tests :(

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.