Add an Sequence object to the decoders

[SaulLu]

I wonder if it would be useful to have a sequence object for the decoders too.

It seems to me for example that if we build a tokenizer with a BPE model that defines a end_of_word_suffix, we will need to use the BPEDecoder decoder to replace theend_of_word_suffix and if we also used a ByteLevel pre-tokenization we will need the ByteLevel decoder to realign the codes.

At the moment, it seems to me that we don't have a solution to choose a suitable decoder for such a tokenizer.

What do you think? 😄

[Narsil]

Same as #873.
I am very favorable in principle.

def decode(tokens: List[str]) -> str

Is destructive, so we need to make it non destructive first.

It seems however that for this one the path seems simpler, we would need to check every single one but it seems decoders could be changed into

def decode(tokens: List[str]) -> List[str]

and make the final pass that is always the same ("".join(tokens)).

If that assumption holds, I guess it's very doable.

[SaulLu]

This is great news!

Your diagnosis makes a lot of sense (after I'm not super familiar with the code base, but I imagine that with tests we will confirm this diagnosis).

[SaulLu]

To transcribe a design that had been discussed offline a long time ago.

It was discussed to implement 2 methods for each decoder: 1) keep the current decode method (to be backward compatible) and 2) implement a new method e.g. decode_for_chain which returns a list. This way the decode method of decoders.Sequence will apply all decode_for_chain methods in chain of the listed decoders and a final chaining.

[SaulLu]

Reopening due to ec43947