[User] ChatML Support

[zhibor]

Is it on the roadmap to support ChatML proposed by openai?

[staviq]

Not officially ( I don't think so ), but I was planning on giving it a go after #3538 because it enables proper control/special token usage in prompt "templates".

Some kind of very simple per model prompt template / library system would be nice to have, I just need to figure out a way to do it cleanly and non-invasively.

[shibe2]

server kind of supports ChatML almost to the same extent as other formats. That is, server itself does not deal with formats. It is client's job to format text according to model's requirements. For ChatML, the client needs to know identifiers of special tokens. Then it creates the prompt as JSON array with token ids and text fragments. In the example in the documentation, you need to replace {"token": "token name"} with actual token identifiers.

bos = 1
im_start = 32001
im_end = 32002
prompt = [
 bos,
 im_start, "system\nYou are ChatGPT, a large language model trained by OpenAI. Answer as concisely as possible.\nKnowledge cutoff: 2021-09-01\nCurrent date: 2023-03-01", im_end, "\n",
 im_start, "user\nHow are you", im_end, "\n",
 im_start, "assistant\nI am doing well!", im_end, "\n",
 im_start, "user\nHow are you now?", im_end, "\n",
 im_start, "assistant\n",
]

You need to explicitly add BOS if the first thing in the prompt would otherwise be another token identifier, in this case, 32001. The client then passes this prompt to the server, and everything else is as usual.

It would be better if identifiers of special tokens could be queried from the server instead of having them in client code.

[xoich]

That makes sense. But are the numbers (1, 32001, 32002) equal for all models? If not, how do you get them?

[staviq]

I don't think there is any actual hard standard for token ids

[shibe2]

BOS is usually 1. Others are usually in a file like added_tokens.json.

I use a patch that allows querying some of special tokens, like BOS, but not yet added tokens.

[DenisSergeevitch]

@zhibor it is merged to the main now with --chatml flag