Need to add CFG Scale and CFG and context free grammars

[ArEnSc]

No description provided.

[amakropoulos]

Thanks for the issue!
I think you mean adding the following options from llama.cpp, is it correct?
--grammar / --grammar-file
--cfg-negative-prompt / --cfg-negative-prompt-file
--cfg-scale

If yes, sure I can do that.
I'm trying to think of a good way to add different options from llama.cpp without blowing up the UI.
Maybe I would separate some options that are probably less frequently in a usual chat setting.
and provide a field that the used can add arguments to the llama.cpp server command.
What do you think?

[ArEnSc]

yeah I wonder if this is supported onthe llamafile server, I checked I don't see those last 3 options

[amakropoulos]

I'm upgrading to llamafile 0.6, that contains those :)

[ArEnSc]

I'm upgrading to llamafile 0.6, that contains those :)

amazing! =D

[amakropoulos]

I have released v1.0.5 that allows grammars along with other parameters.
Unfortunately I can't set the negative-prompt and the scale because they are not exposed as arguments from the llama.cpp server :/

[ArEnSc]

I have released v1.0.5 that allows grammars along with other parameters. Unfortunately I can't set the negative-prompt and the scale because they are not exposed as arguments from the llama.cpp server :/

yeah it was confusing since I saw args code that implied that it was availible from llamafile, I think ill or someone will need to go in there and make the changes to the server and understand how the CFG scale works once my project is off the ground, amazing work though btw! =)
from my understanding CFG scale requires that we run inference twice and the server did drift away from the original PR made and it was never merged

[amakropoulos]

yes exactly, I saw the llamafile args and confused it 🙈

[amakropoulos]

Old topic, but I revisited it :).
The llama.cpp server does not support CFG at the moment, only the main example does.
There is a WIP PR here: ggerganov/llama.cpp#2217