Remove "first token must be BOS" restriction #2153
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, the
llama_eval_internal
function requires the first token in thetokens
array to be a BOS token (=1).I believe that this is not necessary, as
In other words, the BOS token adds a "beginning of document" bias that can be optionally removed.
I originally encountered the "first token must be BOS" error while trying to evaluate llama.cpp using a transformers wrapper that I am working on here. The evaluation fails because the first token in the sequence provided by my code, which is based on this tutorial, is not BOS.