Add ability to instantiate tokenizers from the Hub (from_pretrained)

[n1t0]

Similarly to what exists in transformers, add the ability to instantiate tokenizers from tokenizer.json files uploaded to the Hugging Face Hub.

Example:

from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained('bert-base-uncased')

[Narsil]

Shouldn't we add 1 test per binding + 1 for rust at least ?
It will depend on network+hub which is never great but I think it could be helpful.

Also we probably need to add it to the docs, no ?

[Narsil]

Shouldn't this be "tokenizers", "rust" or something by default ?

[n1t0]

It is, by default the user agent always contains tokenizers/{RUST_VERSION}, this is for additional items

[n1t0]

Shouldn't we add 1 test per binding + 1 for rust at least ?
It will depend on network+hub which is never great but I think it could be helpful.

Agreed

Also we probably need to add it to the docs, no ?

I've already added it in every place I though about. Do you have something specific in mind?

[Narsil]

I've already added it in every place I though about. Do you have something specific in mind?

I missed it in quicktour that was what I had in mind.