Mechanism for storing prompt templates

[simonw]

This will allow users to store templates for complex prompts (both as system prompts and regular prompts that have strings interpolated into them) so they can use them in the future.

llm templates edit summary
# An editor opens to edit that prompt
llm --template summary "$(curl -s https://www.example.com/)"

With a shortcut so llm -t summary works too.

[simonw]

The trickiest thing about these templates is their design.

The simplest form of template is some text with a replacement variable where the user's additional input is pasted in - something like this:

Summarize this: {input}

But there are all sorts of other things to consider:

• How should system prompts be handled? I'd much rather write the above template as a system prompt of Summarize this with the input being sent as the full regular prompt.
• How to support more than one placeholder? Would be neat if you could say llm -t persona -p name Tom -p age 22
• If a template has both a prompt and a system prompt, how are they both stored?

I'm leaning towards YAML for this, because it has neat support for multi-line text blocks.

[simonw]

Templates can go in ~/Library/Application Support/io.datasette.llm/templates - each one can be a file.

Will I force them to be YAML files, or can you have one that is pure text if it's a really simple one?

[simonw]

I think YAML will do. A YAML file can contain just a string, and it will be treated right unless it happens to contain a colon character or similar:

>>> import yaml
>>> yaml.safe_load("this is just a string")
'this is just a string'
>>> yaml.safe_load("this is just a string: hooray")
{'this is just a string': 'hooray'}
>>> yaml.safe_load("\"this is just a string: hooray\"")
'this is just a string: hooray'

I can explain enough of this in the documentation to avoid people getting caught out.

[simonw]

What should I use for the actual variable substitution part of this? I'd like to keep that to the Python standard library if possible.

I'm tempted to just use this:

>>> s = "Summarize: {input}"
>>> s.format(blah='foo')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'input'
>>> s.format(input='foo')
'Summarize: foo'

Another option is Template strings: https://docs.python.org/3/library/string.html#template-strings

Template strings support $-based substitutions, using the following rules:

• $$ is an escape; it is replaced with a single $.
• $identifier names a substitution placeholder matching a mapping key of "identifier". By default, "identifier" is restricted to any case-insensitive ASCII alphanumeric string (including underscores) that starts with an underscore or ASCII letter. The first non-identifier character after the $ character terminates this placeholder specification.
• ${identifier} is equivalent to $identifier. It is required when valid identifier characters follow the placeholder but are not part of the placeholder, such as "${noun}ification".

>>> from string import Template
>>> t = Template("Summarize: $input")
>>> t.substitute(input="Hello")
'Summarize: Hello'
>>> t.substitute(input2="Hello")
Traceback (most recent call last):
...
KeyError: 'input'

[simonw]

OK, I think I'm going to go with $variable substitution using string.Template - and the templates themselves will be YAML files which can contain either a string or a dictionary.

If it's a dictionary it can have prompt: and system: keys for prompt + system templates.

If you just have a system then the prompt will default to $input.

Any other $name variables will be treated as required parameters, passed using -p name value.

[simonw]

So a set of commands:

llm templates edit name-of-template # Edit the specified template, using $EDITOR
llm templates list # list available templates
llm templates show name-of-template # show a template
llm templates path # path to the templates directory

[simonw]

Templates can also have a model: ... to set the default model for that template.

[simonw]

I can use this: https://click.palletsprojects.com/en/8.1.x/utils/#launching-editors

>>> import click
>>> click.edit(filename='/tmp/hello.txt')

[simonw]

Idea from:

• Ability to mess with logit biases #36

Slight twist: what if you want to use another template at the same time?

Might be an argument for supporting combined templates - pass -t multiple times and the result is a combination of those templates - the most recent of each of the prompt, system prompt, logit biases etc.

[simonw]

It would be amazing if you could install new templates by installing Python packages - using a plugin hook. Could be a neat way to distribute more follows templates, especially ones that include functions.