Improper Handling of Guillemets (»«) cause hallucinations

[clemcer]

I believe this type of quotaton mark (»«) seems to trigger hallucinations somewhat consistently.
At least in German but probably other languages as well.
This type of quotation mark (»«), known as Guillemets or Chevrons, is not commonly used in English, at least ChatGPT told me so lol, so maybe the code doesnt account for them?

Here is an example video, the red lines represent hallucinations.

hallucinations.mp4

Here is the text file:
german_hallucinations_1.txt

[ROBERT-MCDOWELL]

no, it's not the double quotes making hallucinations, it's more the double quotes and the whole empty space + new lines making troubles...

[ROBERT-MCDOWELL]

could you provide the real original text so I can check if it's space, tabs or else....
usually the doubl quotation marks are opening, then text then closing. In German it seems the opposite right? or it's another fancy book where they decided to play with the quotation marks?

[clemcer]

I edited my original post and added the text file.

And I made another video from the text I uploaded on another issue.

german_hallucinations_2.mp4

If you want only the text visible in the video here it is:
german_hallucinations_2.txt

usually the doubl quotation marks are opening, then text then closing. In German it seems the opposite right? or it's another fancy book where they decided to play with the quotation marks?

No, this type of quotation mark is pretty common in German, if not standard, at least for novels. I went through my ebooks and havent found one where these quotation marks aren't being used.

And just as a heads up I even found one where they are reversed:

«Passen Sie auf, wo Sie hintreten, Dr. Hunter»,

edit:
I kind of read over that part

usually the doubl quotation marks are opening, then text then closing

and thought you were asking about the use of this type of quotation mark in general but to give a more precise answer: In German these quotation mark are almost exclusively used with their tips pointing inward like this (»«} instead of («»} although I have found one ebook where they are being used like this («»}. (but that is definetly the exception).

[ROBERT-MCDOWELL]

ok I'm going to make some tests to see if my patch works