Option to turn token integers back into text

[simonw]

The opposite of this:

echo "Show these tokens" | ttok --tokens
# Outputs: 7968 1521 11460 198

[simonw]

Potential names for this:

• ttok --text (since it is the opposite of --tokens)
• ttok --to-text
• ttok --decode

I think I like --decode the most. It maps to the underlying .decode() method.

I could add --encode as an alias for --tokens for added consistency.

[simonw]

This should support space separated, comma separated and JSON arrays or integers.

I think I'll just use a \d+ regular expressions to parse integers out of the input.

[simonw]

Got GPT-4 to write this: https://chat.openai.com/share/a3c5da38-bfd0-423d-af7e-dbed7bfe5278

@click.option("--decode", "decode", is_flag=True, help="Decode token integers to text")

# ...

    if decode:
        # Use regex to find all integers in the input text
        tokens = [int(t) for t in re.findall(r'\d+', text)]
        decoded_text = encoding.decode(tokens)
        click.echo(decoded_text)

[simonw]

I needed this to help test:

• Ability to mess with logit biases llm#36

[simonw]

Oops, did that work on the wrong branch.

[simonw]

$ ttok --tokens show me the tokens
3528 757 279 11460
$ ttok --encode show me the tokens
3528 757 279 11460
$ ttok --decode 3528 757 279 11460
show me the tokens