Wrong tokenization of "Shell" as "She, ll"

[vikrantsharma7]

nlp = spacy.load("en")
doc = nlp(u"Shell is a good brand.")
pretty_print(doc)  # custom function to print dependency tree

gives the following output:

is (VBZ) ROOT
l ---   ll (MD) nsubj
l ------   She (PRP) nsubj
r ---   brand (NN) attr
l ------   a (DT) det
l ------   good (JJ) amod
r ---   . (.) punct

"Shell" should not be tokenized into ["She", "ll"].
Could not even find this mapping in tokenizer/specials.json

Any way to fix this?

• Operating System: Ubuntu 16.04 LTS
• Python Version Used: 2.7.12
• spaCy Version Used: 1.6.0

[ines]

Ah, damn, this should have be added to the excluded tokenizer exceptions in spacy/en/tokenizer_exceptions.py. Will fix this right now and add a regression test.

The specials.json is the old, deprecated data btw. It's still in use in the current models, but this will change with the v2.0 release to keep things more consistent.

[vikrantsharma7]

Thanks! :)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.