[FEATURE] Built-in tokenization feature for Text objects

[mynameisvinn]

🚨🚨 Feature Request

• Related to an existing Issue
• A new implementation (Improvement, Extension)
  Text objects should have a built-in method to convert strings to tokens, according to a user-specified tokenizer.

Is your feature request related to a problem?

Partially. Text objects currently use Bert tokenizers by default.

If your feature will improve HUB

A common use case is streaming data from a Hub Dataset to a Hugging Face transformer. This feature would remove friction from that process.

Description of the possible solution

A solution would improve the current syntax:

ds = hub.Dataset(tag, shape=(10,), schema=schema, mode="w", tokenizer=some_tokenizer) 

[mynameisvinn]

Closing this feature request due to inactivity and lack of interest. Will revive it if more users request it.