Usage examples (pypi and readme) are outdated [BPETokenizer rename]

[tsteffek]

Edit: BPETokenizer has been renamed to CharBPETokenizer, however, all the examples still feature the BPETokenizer.

Original:
I just tried using the pypi package and it didn't work.

I opened a new virtual environment, installed tokenizers versions 0.4.2, 0.4.1, 0.4.0 and 0.3.0, started a Python console (using PyCharm, to be exact) and typed: from tokenizers import BPETokenizer.
For all these versions it would tell me:

Traceback (most recent call last):
  File "<input>", line 1, in <module>
ImportError: cannot import name 'BPETokenizer' from 'tokenizers' (C:\tools\miniconda3\envs\tokenizer\lib\site-packages\tokenizers\__init__.py)

Tokenizers 0.2.1 works fine.

---

My machine:

• Win10 x64 Version 10.0.18363 Build 18363
• Python 3.7.6
• miniconda 4.8.0
• pip 20.0.2

[amoux]

I got the same issue in the latest version. But after installing version 0.2.1 I had no errors importing.

• Ubuntu 19.10
• Python 3.7 Anaconda

(python-37) user@user:~$ ipython
Python 3.7.6 | packaged by conda-forge | (default, Jan  7 2020, 22:33:48) 
In [1]: from tokenizers import BPETokenizer                                                             
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-812973393cae> in <module>
----> 1 from tokenizers import BPETokenizer
ImportError: cannot import name 'BPETokenizer' from 'tokenizers' (/home/user/anaconda3/envs/python-37/lib/python3.7/site-packages/tokenizers/__init__.py)

[aditya-malte]

Facing the same issue

[tsteffek]

Ok, just looked at the changelogs, v0.3.0 did:

• BPETokenizer has been renamed to CharBPETokenizer for clarity.

So, obviously importing BPETokenizer doesn't work. I'll change the name of the issue then...