Add test for MPT tokenization

[goerch]

Unfortunately I don't see a way to distinguish CONTROL and USER_DEFINED tokens when using AutoTokenizer.

[Galunid]

I adapted changes from this PR locally on my stablelm branch and they seem to fix the issue from #3604 .

When passing The best music is as a prompt (note two spaces at the end), they seem to get stripped though
The best music isthe music you have the most fun making.

[goerch]

When passing The best music is as a prompt (note two spaces at the end), they seem to get stripped though
The best music isthe music you have the most fun making.

I think this is what I feared: there are special and non-special added tokens which have to be handled before the core tokenizer's encode and decode and we can't distinguish them currently.

Edit: @jploski just showed us a way to distinguish them via 74204cc#commitcomment-130628610. So I tend to merge this PR and improve on handling the mentioned issue in another one.

[Galunid]

I just wanted to let you know, for stablelm use case it's fine, (added_vocab is only spaces and end of text). Thanks for this!

[goerch]

I adapted changes from this PR locally on my stablelm branch and they seem to fix the issue from #3604 .

When passing The best music is as a prompt (note two spaces at the end), they seem to get stripped though The best music isthe music you have the most fun making.

To fix this issue I imagine to change the detokenizer code to output USER_DEFINED tokens (this should repair the problem) and adapt the test cases to ignore USER_DEFINED tokens where needed. Will propose this once the test cases are done. Does this sound like a plan?