Transformers breaks when running setup.py

[AdamLouly]

System Info

Running python setup.py install in transformers folder throws this error:

error: huggingface-hub 0.17.2 is installed but huggingface_hub<0.17,>=0.16.4 is required by {'tokenizers'}

This starts happening after merging this PR where line 175 was changed from:
from "tokenizers>=0.11.1,!=0.11.3,<0.14",
to "tokenizers>=0.14,<0.15",

#23909

when I change it back manually it works fine.

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Running python setup.py install in transformers folder

Expected behavior

error: huggingface-hub 0.17.2 is installed but huggingface_hub<0.17,>=0.16.4 is required by {'tokenizers'}

[ArthurZucker]

A quick fix: update huggingface_hub . On our side we should update the dependency requirement. Thanks for reporting!

[AdamLouly]

@ArthurZucker have you tested re installing Transformers after this update?
I tested and its still failing.

huggingface-hub>=0.16.4,<1.0 won't solve the issue:
huggingface_hub<0.17,>=0.16.4 is required by {'tokenizers'}

because its installing huggingface_hub 0.17.4 and that will break it.

[ArthurZucker]

I think tokenizers has to be updated at this point, since it's not using the latest dependency. cc @Narsil any reason why we limited to HF hub 0.17?

[AdamLouly]

@ArthurZucker Any updates on this?

[ArthurZucker]

Yep, this PR was merged in tokenizers, we just need a release and this will be adressed