Reprocessing Issue with Llama 3

[Nabokov86]

When using Llama 3, I've noticed that unnecessary reprocessing occurs on previously generated text.
To reproduce this issue, try generating a short piece of text couple of times and see how the processing sometimes happens.

Latest concedo_experimental.

[Nabokov86]

It seems like the reprocessing occurs after a new line is generated.

[LostRuins]

Did you by any chance enable "Trim Sentences" or "Author Note"?

[Nabokov86]

No, I use default settings without trimming. So, you can't reproduce it?
saved_story.json

[LostRuins]

Yes, I can reproduce it. Looking closer, the tokenizer is behaving weirdly. I think there is an issue with token merges.

Relevant: ggerganov#6809

You should experience a small amount of reprocessing all the way back to the previous newline. This is a bug.

[LostRuins]

Hi, Should be fixed in the latest version. Remember to get freshly reconverted GGUFs

[Nabokov86]

@LostRuins Thanks! Yes, it looks like it’s working now. Thank you for continuing to maintain this project, you’re awesome!