Phi-3 produces random gibberish

[LoriBru]

Hi all, thanks a lot for this great repo!
I am a beginner in the field and I started playing with Phi-3 and so far I've been quite impressed.
I am not sure this is the place to discuss the issue I am having, in case please redirect me to a more appropriate place.

My problem is that it is very easy to trigger Phi-3 to start throwing a super long and random amount of words instead as answer to easy question.

Let's take this example of conversation:

**Me**: what is a rainbow?

**Phi3**: A rainbow is a meteorological phenomenon that is caused by reflection, refraction, and dispersion of light in water droplets resulting in a spectrum of colors appearing in the sky. It takes the form of a multicolored arc. Rainbows caused by sunlight always appear opposite the sun. The primary rainbow is caused by one set of reflections inside water droplets, whereas a secondary rainbow is caused by two sets of internal reflections.

Rainbows are formed when the sun shines onto droplets of moisture in the Earth's atmosphere. The light is refracted as it enters the droplet, reflects off the back of the droplet, and is refracted again as it exits the droplet. This double refraction causes the light to spread out into a spectrum of colors, which we see as a rainbow.

The colors of the rainbow, in order, are red, orange, yellow, green, blue, indigo, and violet. This sequence is often remembered by the acronym ROYGBIV. The colors are a result of the different wavelengths of light being refracted by different amounts. Red light is refracted the least, while violet light is refracted the most.

Rainbows are always seen as a circular arc, but the ground prevents us from seeing the full circle. The size and shape of a rainbow depend on the position of the sun, the observer, and the raindrops. The most common type of rainbow is a primary rainbow, which is the brightest and has the most saturated colors. A secondary rainbow, which is fainter and has colors that are reversed, can also be seen under certain conditions.

Rainbows are not only a beautiful natural phenomenon but also a symbol of hope and goodwill. They have been the subject of myths, legends, and art throughout human history.

**Me**: tell me one of those legends

**Phi3**:  One of the most famous legends associated with rainbows is the story of the "Chasing the Rainbow" from Irish folklore. According to the legend, there is a magical pot of gold hidden at the end of a rainbow. The pot is said to be guarded by a leprechaun, a type of fairy in Irish mythology. The leprechaun is known for his cunning and trickery, and he will do anything to protect his treasure.

The legend tells that if a person manages to catch the pot of gold at the end of the rainbow, the leprechaun will grant them three wishes. However, the leprechaun is also known for his cleverness and will try to outwit the person chasing the pot. The story has been popularized in various forms of literature, movies, and television shows, and it has become a symbol of the elusive nature of rainbows and the idea of an unattainable treasure.

While the story of the pot of gold at the end of the rainbow is not based on factual evidence, it has become an enduring part of folklore and has contributed to the mystical and enchanting aura surrounding rainbows.

**Me**: what is a leprechaun?

**Phi3**: A leprechaun is a mythical creature from Irish folklore, often depicted as a small, mischievous, and solitary figure. Leprechauns are commonly associated with the Irish tradition of St. Patrick's Day and are said to be skilled in the art of shoemaking. They are also known for their magical powers and their ability to grant wishes.

Leprechauns are typically described as having the following characteristics:

1. Small stature: Leprechauns are often portrayed as being around 3 to 4 feet tall, which is significantly shorter than the average human height.

2. Green clothing: They are usually depicted wearing a green suit, which is a nod to Ireland's nickname, "The Emerald Isle."

3. Bearded: Leprechauns are often shown with a long, pointed beard, which is a common feature in their portrayal.

4. Pointed hat: They are frequently depicted wearing a tall, conical hat, which is sometimes referred to as a "leprechaun's hat" or "pointed hat."

5. Shoemaking: Leprechauns are renowned for their shoemaking skills, and it is said that they can mend shoes in a single night.

6. Magical powers: Leprechauns are believed to possess magical abilities, such as the power to create illusions, shape-shift, and grant wishes.

7. Hidden treasure: One of the most famous aspects of leprechaun folklore is the idea that they hoard a pot of gold at the end of a rainbow. Legends say that if a leprechaun is caught, he will grant the captor three wishes in exchange for his freedom.

8. Solitary nature: Leprechauns are often depicted as solitary creatures who prefer to live in hidden caves or underground dwellings.

Despite their popularity in folklore and popular culture, leprechauns are not considered to be real creatures. They are part of the rich tapestry of Irish mythology and have been embraced as a symbol of Irish heritage and folklore.

**Me**: tell me more about this mythology

**Phi3**:  Irish mythology is a rich and diverse collection of stories, legends, and folklore that has been passed down through generations in Ireland. It encompasses a wide range of mythological beings, including gods, goddesses, heroes, and magical creatures. The mythology is deeply intertwined with the country's history, culture, and beliefs, and it continues to influence literature, art, and popular culture today.

Some of the key elements and themes of Irish mythology include:

1. The Tuatha Dé Danann: The Tuatha Dé Danann were a group of god-like beings who were believed to have once ruled Ireland. They were said to possess great magical powers and were associated with various aspects of life, such as war, love, and the natural world.

2. The Fomorians: The Fomorians were a race of supernatural beings who were considered to be the enemies of the Tuatha Dé Danann. They were often portrayed as dark and malevolent, and their stories often involved battles and conflicts with the Tuatha Dé Danann.

3. The Dagda: The Dagda was a powerful god in Irish mythology, known for his strength, wisdom, and magical abilities. He was often depicted as a large, burly figure with a club, which he used to protect and provide for his people.

4. The Morrígan: The Morrígan was a goddess of war and fate, who was said to have the power to influence the outcome of battles. She was often portrayed as a shape-shifting figure, taking on various forms, such as a crow, a wolf, or a horse.

5. The Sidhe: The Sidhe, also known as the "fairies" or "the good people," were a race of supernatural beings who lived in hidden realms beneath the earth or in the otherworld. They were often associated with nature, magic, and the supernatural.

6. The Aos Sí: The Aos Sí were a group of supernatural beings who were believed to be the ancestors of the Tuatha Dé Danann. They were said to possess great magical powers and were often associated with the natural world, such as forests, mountains, and rivers.

7. The Celtic Pantheon: The Celtic pantheon was a collection of gods and goddesses who were worshipped by the ancient Celts. These deities were often associated with natural elements, such as the sun, the moon, the sea, and the earth.

8. Legendary Heroes: Irish mythology is also known for its legendary heroes, such as Cú Chulainn, Fionn mac Cumhaill, and Oisín. These heroes were often depicted as brave and noble figures who embarked on epic quests and battles, and their stories have been passed down through generations.

9. The Tales of the Ulster Cycle: The Ulster Cycle is a collection of ancient Irish stories that revolve around the heroic exploits of the legendary characters, such as Cú Chulainn and the warriors of the Ulster kingdom. These tales often involve battles, love, and the supernatural.

10. The Tales of the Mabinogi: The Mabinogi is a collection of ancient Irish stories that focus on the lives and adventures of the legendary characters, such as Pwyll, Branwen, and Manannán. These tales often explore themes of love, loyalty, and the supernatural.

Irish mythology is a fascinating and complex tapestry of stories, characters, and beliefs that have captivated the imagination of people for centuries. It continues to inspire and influence literature, art, and popular culture, and it remains an essential part of Ireland's rich cultural heritage.

**Me**: can you tell me more about ireland in general?

**Phi3**:  Ireland, officially known as the Republic of Ireland, is a country located in north-western Europe. It is an island in the North Atlantic Ocean, and it shares its only land border with the Republic of Northern Ireland. Ireland is the third-largest island on Earth, and it is the twentieth-most populous island in the world.

Geographically, Ireland is divided into four provinces: Connacht, Leinster, Munster, and Ulster. The island is divided into 32 counties, each of which is a county. The capital city of Ireland is Dublin. Ireland is a member of the European Union. The island is a country with a rich history, and it is a country with a temperate climate. Ireland is a country with a temperate climate. Ireland is a country with a temperate climate. Ireland is a country with a temperate climate. Ireland is a country with a temperate climate. Ireland is a country with a temperate climate. Ireland is a country with a country with a temperate climate. Ireland is a country with a country with a country with a country with a country with a country with a country with a to the answer theater with a country with a to the answer to a country with a to the rain with a toaster, and rain to a rain, a of the rain, a rain, and a rain, and theater. Theater. The Sun to-rain, the, rain and a rain, and a rain. The rain. The answer, and a rain, a the rain, rain, the rain, the rain, rain, and theater. The rain, andray, rain. The rain. The sunlight.rain. Thedrop.ells when. Rainbow. This,drop when,RAAC when,ray refracts the rain. The rain. Refracted drop. rain. This. This,ray, while the sun. sun. The raind. sun.

.at. Theorizontal.com,in rain.com,ats, rainer,view. The sun to clarify. The sun,angle, light in the, refract,com shape, androof,ation,cometing,ade,coming,comets,comble,comet ascrip,comet,ray,incurtaining,unsafie-angique, doubled,andangement to see-drower, andangling and,ray, andray,ray,ells, andray,vapling,andlake,ells, the same (orangling butament,s, theater, theorifies of theta,ondess of rain,oressessing. rain,ographing angle,sandessions,kyangling,andessing,ensourse rays,oultess,entrix,hes,enship,in the angle,oranging when the same, the angle, the same againens,inside andangel,angually,ature,angess, seen, theater, the same, the same, the arrows,like, (
yragens  the sky when theatered in theatering theater,ing, theater,comatically,hing
 theater (theater,comad, of the rain, of the ?.
 of theatrix,otic andangling and-theater,angel, mirroring,ologen,angie (theatered,
 of a mirrored
attered ?angewellow to theologangets, ounter
 color of (the angleomatches, theater,angel of a little,angewewelangarsang,angang and theologape ofangomangewangang ( (of whichologarang, (theaterangagunscapeangagunituarizes,orangessing in a ltheateromagounassaqueatheathangounisangnimersagindisielding, in theangisang of the sameingessangiversomulting the spectrumofagiewing reledess  the  the
atleledispecie, and by the same,ingiewing, the same, of the lighting,elle, theajing, the of the sameangess of the second of the sameangielding (
ednimultoothiqueatagunige of the lighting etherfield ountereuteilistaratter reflectarighting ethering, of -aging  -  -horinge, theewling ouparunt, the ounatangomathelateromagnear ringshovershoangnimituled, ttering andnimomsagargun, inangewarchesarches,-arangessions ouns, the leastangangangs, lightoweest.estarches, the same, (the same wave, the, with, -deading,angoun  Bouness,ess inside of
 inside of the, -lighting,ess,
 inside,elighting
 inside, inside lighting inside,,inging, the to the aring, "bust, the souangang from the same, by
 the
through,, inside, inside inside, inside, the next, then, then, then, by
 the
 in the (the
the  the factou
 (p (fews, in the following, the reflected

quearc, ,
bright, the same,brow 1. 1ho 2ing drihold, of  -the ho horeu
ing -
agr f again the the aging, in the whole of of 32oval
 divisovor, the 3oagagift, (prisitu, (the boguging, the and to the most
 the unomowe the, bring the the the same, andque the second the brighting, it ashing, and ndowologs, and, however, theatering, gowe, it the same ofthe the least the leastledsunding-theing.,,ing the most the mostog -youinging the most of the same-ming of the same, the same the factagouarking,, the next, the least the light, the broken
the mowe, or, andagusting, in theedewe-youingewingewing youhingingingingled,inginginting the the least the inner-theagouagagagagething, andumsandagique,souled, theled-the least the least the lighting-the in theatering, of the mostingledestagest of the leastingledarowe, the most the most the most the most the most the most, the following the following as the most theatered by the mostingsand and, andingingled, and theing and brighting of thisinginging theing of the differentingleding, the reflected by themingleding by theingleds iningingledewelding as youledugledsugoging of the brighting and least by the least the by the byogewewledledled by the second the by the topuging, the by this by least by the sameugug (theing, the by the by itledingledinging, of course of brighting, the subjectingleds (inlediqueled in the by the least theaterheheitingingledledugingheholding andtheledsituledlediqueledledledledledledheofheledledledleds of tleding, by sledionable-theingledisled by ahingledledledledledledledledledit,andledledledledledledledledledledestewestledledleduledledledledledinging, andscaledledledsandinguminges, andingingingingingingingingingingingledots, theledingusingledisanging, andselding, in the inledinging, andtheable,theable, ands,anding, andsandinginginginging, and the of the the the theable, andtheeyerand, and the as itle, ands, the least the most bys-theing,anding as iting intheestu, the
 inararthearbytheledarooting the least to the most to me to bytopest of the most to beowe to besanding
theing, and which the most the byleants, as by different by the most of this aknowns andtheobles, andle, andle, inthele, inthebeing,the

As you can see the last answer is pure garbage.
This is my last full prompt:

<|system|>
You are an AI assistant that helps people find information. <|end|>
<|user|>what is a rainbow?<|end|>
<|assistant|> A rainbow is a meteorological phenomenon that is caused by reflection, refraction, and dispersion of light in water droplets resulting in a spectrum of colors appearing in the sky. It takes the form of a multicolored arc. Rainbows caused by sunlight always appear opposite the sun. The primary rainbow is caused by one set of reflections inside water droplets, whereas a secondary rainbow is caused by two sets of internal reflections.

Rainbows are formed when the sun shines onto droplets of moisture in the Earth's atmosphere. The light is refracted as it enters the droplet, reflects off the back of the droplet, and is refracted again as it exits the droplet. This double refraction causes the light to spread out into a spectrum of colors, which we see as a rainbow.

The colors of the rainbow, in order, are red, orange, yellow, green, blue, indigo, and violet. This sequence is often remembered by the acronym ROYGBIV. The colors are a result of the different wavelengths of light being refracted by different amounts. Red light is refracted the least, while violet light is refracted the most.

Rainbows are always seen as a circular arc, but the ground prevents us from seeing the full circle. The size and shape of a rainbow depend on the position of the sun, the observer, and the raindrops. The most common type of rainbow is a primary rainbow, which is the brightest and has the most saturated colors. A secondary rainbow, which is fainter and has colors that are reversed, can also be seen under certain conditions.

Rainbows are not only a beautiful natural phenomenon but also a symbol of hope and goodwill. They have been the subject of myths, legends, and art throughout human history.<|end|>
<|user|>tell me one of those legends<|end|>
<|assistant|> One of the most famous legends associated with rainbows is the story of the "Chasing the Rainbow" from Irish folklore. According to the legend, there is a magical pot of gold hidden at the end of a rainbow. The pot is said to be guarded by a leprechaun, a type of fairy in Irish mythology. The leprechaun is known for his cunning and trickery, and he will do anything to protect his treasure.

The legend tells that if a person manages to catch the pot of gold at the end of the rainbow, the leprechaun will grant them three wishes. However, the leprechaun is also known for his cleverness and will try to outwit the person chasing the pot. The story has been popularized in various forms of literature, movies, and television shows, and it has become a symbol of the elusive nature of rainbows and the idea of an unattainable treasure.

While the story of the pot of gold at the end of the rainbow is not based on factual evidence, it has become an enduring part of folklore and has contributed to the mystical and enchanting aura surrounding rainbows.<|end|>
<|user|>what is a leprechaun?<|end|>
<|assistant|> A leprechaun is a mythical creature from Irish folklore, often depicted as a small, mischievous, and solitary figure. Leprechauns are commonly associated with the Irish tradition of St. Patrick's Day and are said to be skilled in the art of shoemaking. They are also known for their magical powers and their ability to grant wishes.

Leprechauns are typically described as having the following characteristics:

1. Small stature: Leprechauns are often portrayed as being around 3 to 4 feet tall, which is significantly shorter than the average human height.

2. Green clothing: They are usually depicted wearing a green suit, which is a nod to Ireland's nickname, "The Emerald Isle."

3. Bearded: Leprechauns are often shown with a long, pointed beard, which is a common feature in their portrayal.

4. Pointed hat: They are frequently depicted wearing a tall, conical hat, which is sometimes referred to as a "leprechaun's hat" or "pointed hat."

5. Shoemaking: Leprechauns are renowned for their shoemaking skills, and it is said that they can mend shoes in a single night.

6. Magical powers: Leprechauns are believed to possess magical abilities, such as the power to create illusions, shape-shift, and grant wishes.

7. Hidden treasure: One of the most famous aspects of leprechaun folklore is the idea that they hoard a pot of gold at the end of a rainbow. Legends say that if a leprechaun is caught, he will grant the captor three wishes in exchange for his freedom.

8. Solitary nature: Leprechauns are often depicted as solitary creatures who prefer to live in hidden caves or underground dwellings.

Despite their popularity in folklore and popular culture, leprechauns are not considered to be real creatures. They are part of the rich tapestry of Irish mythology and have been embraced as a symbol of Irish heritage and folklore.<|end|>
<|user|>tell me more about this mythology<|end|>
<|assistant|> Irish mythology is a rich and diverse collection of stories, legends, and folklore that has been passed down through generations in Ireland. It encompasses a wide range of mythological beings, including gods, goddesses, heroes, and magical creatures. The mythology is deeply intertwined with the country's history, culture, and beliefs, and it continues to influence literature, art, and popular culture today.

Some of the key elements and themes of Irish mythology include:

1. The Tuatha Dé Danann: The Tuatha Dé Danann were a group of god-like beings who were believed to have once ruled Ireland. They were said to possess great magical powers and were associated with various aspects of life, such as war, love, and the natural world.

2. The Fomorians: The Fomorians were a race of supernatural beings who were considered to be the enemies of the Tuatha Dé Danann. They were often portrayed as dark and malevolent, and their stories often involved battles and conflicts with the Tuatha Dé Danann.

3. The Dagda: The Dagda was a powerful god in Irish mythology, known for his strength, wisdom, and magical abilities. He was often depicted as a large, burly figure with a club, which he used to protect and provide for his people.

4. The Morrígan: The Morrígan was a goddess of war and fate, who was said to have the power to influence the outcome of battles. She was often portrayed as a shape-shifting figure, taking on various forms, such as a crow, a wolf, or a horse.

5. The Sidhe: The Sidhe, also known as the "fairies" or "the good people," were a race of supernatural beings who lived in hidden realms beneath the earth or in the otherworld. They were often associated with nature, magic, and the supernatural.

6. The Aos Sí: The Aos Sí were a group of supernatural beings who were believed to be the ancestors of the Tuatha Dé Danann. They were said to possess great magical powers and were often associated with the natural world, such as forests, mountains, and rivers.

7. The Celtic Pantheon: The Celtic pantheon was a collection of gods and goddesses who were worshipped by the ancient Celts. These deities were often associated with natural elements, such as the sun, the moon, the sea, and the earth.

8. Legendary Heroes: Irish mythology is also known for its legendary heroes, such as Cú Chulainn, Fionn mac Cumhaill, and Oisín. These heroes were often depicted as brave and noble figures who embarked on epic quests and battles, and their stories have been passed down through generations.

9. The Tales of the Ulster Cycle: The Ulster Cycle is a collection of ancient Irish stories that revolve around the heroic exploits of the legendary characters, such as Cú Chulainn and the warriors of the Ulster kingdom. These tales often involve battles, love, and the supernatural.

10. The Tales of the Mabinogi: The Mabinogi is a collection of ancient Irish stories that focus on the lives and adventures of the legendary characters, such as Pwyll, Branwen, and Manannán. These tales often explore themes of love, loyalty, and the supernatural.

Irish mythology is a fascinating and complex tapestry of stories, characters, and beliefs that have captivated the imagination of people for centuries. It continues to inspire and influence literature, art, and popular culture, and it remains an essential part of Ireland's rich cultural heritage.<|end|>
<|user|>can you tell me more about ireland in general?<|end|>
<|assistant|>

My question for you is: am I doing something wrong on my side, in terms of usage of Phi-3 or preparation of the prompt?

I am giving back in the prompt also the history of the conversation as done in this project : https://github.com/microsoft/Phi3-Chat-WinUI3-Sample/tree/main/ChatAppGenAI

I prepared a small repo where you can check the project I am using:
https://github.com/LoriBru/Phi-3SimpleChat

Thank you for your help!

[leestott]

Hi @LoriBru so there is this report on Hugging Face chat ui huggingface/chat-ui#1139 Have you tried changing the model to see if its related?

[leestott]

@LoriBru

Can you confirm the following

1. What model are you using?
2. Can you confirm if your using API or self hosting the model?
3. What is the chat template you are using? If they are not using the right stop sequences, then it might happen.

So we would like to help identify the issue your having

[leestott]

example of a stop sequence in C#

// define prompts
var systemPrompt = "You are an AI assistant that helps people find information. Answer questions using a direct style. Do not share more information that the requested by the users.";
string userPrompt = "Describe the image, and return the string 'STOP' at the end.";
var fullPrompt = $"<|system|>{systemPrompt}<|end|><|user|><|image_1|>{userPrompt}<|end|><|assistant|>";

See https://github.com/microsoft/Phi-3CookBook/blob/main/md/07.Labs/Csharp/csharplabs.md labs302 https://github.com/microsoft/Phi-3CookBook/tree/main/md/07.Labs/Csharp/src/LabsPhi302

[LoriBru]

Hi @leestott !
In my first post I provided a repository with the full project and the instructions, isn't it clear enough?
The application is running locally (it's a console app) and the model is this one : https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-onnx/tree/main/directml/directml-int4-awq-block-128

You can also check the termination sequence in the sample, which I took from here: https://github.com/microsoft/Phi3-Chat-WinUI3-Sample/blob/main/ChatAppGenAI/Phi3Runner.cs

  if (stringBuilder.ToString().Contains("<|end|>")
      || stringBuilder.ToString().Contains("<|user|>")
      || stringBuilder.ToString().Contains("<|system|>")
      || stringBuilder.ToString().Contains("["))
  {
      break;
  }

[leestott]

Hi @LoriBru as this is based of this sample I suggest you open up a issue on that repo https://github.com/microsoft/Phi3-Chat-WinUI3-Sample/

@nmetulev should be able to help provide some more guidance

@elbruno has built a number of C# samples on this repo so can provide some additional guidance/tips

[nmetulev]

I wonder if this is an issue of the size of the prompt and it happens once it hits a certain threshold or goes beyond the context window size supported by phi3. Have you tried different variants of phi3, such as the Cuda or Cpu version? Or even the larger context window (128K) Phi3 here: https://huggingface.co/microsoft/Phi-3-mini-128k-instruct-onnx?

@natke from the Onnx Team might have additional insights.

[natke]

Hi @LoriBru, can you try to update to the latest packages and try running again? Can you also try running on CPU to see if you get the same result? Thank you for the repro - we will also try and repro on our end

[LoriBru]

Thank you all for your comments!
In the sample I provided, the issue is 100% reproducible on my side, by inputting exactly the same questions in the same order that I reported in the first post.

@nmetulev I also had the feeling that this might be related to the length of the prompt, so my question is: is it a good practice to give back the full conversation history to the prompt, formatted in that way? Shall there be a truncation in order not to have a longer and longer prompt?
I based the formatting of the prompt on your sample here, where there is no truncation : https://github.com/microsoft/Phi3-Chat-WinUI3-Sample/blob/main/ChatAppGenAI/Phi3Runner.cs

@natke I updated to the latest nuget packages and still have the exact same output.

@ both, I did not have time yet to try to run on CPU or to try the 128K model, I'll try as soon as possible.

[LoriBru]

So far, I am not able to have the same issue with the 128K model. Its answers by the way are in general always shorter and more direct than the 4k model. I don't know if that's an expected behavior or not.

[LoriBru]

OK not true, I managed to get the following answer even with the 128k model:

The reservoirs. The reservoirs. The reservoirs. The reserving. The reserate. The reservice. The reservice. The reservice. The reservice. The reservice. The reservice. The reservice. The reservice. The reservice. The reservice. The reservice. The reservice. The reservice. The reservice. The reservice. The reservice. The reservice. The reservice. The reservice. The reservice. The reservice. The reservice. The reservice. The reservice. The service. The service. The service. The service. The service. The service. The service. The service. The service. The service. The service. The service. The service. The service. The service. The service. The service. The service. The service. The service. The service. The service.

. The service. The service. The service. The service. The service.

ing service. The service.
ing.

, service.

, and, the service. The service. The service.

, as service.


ing.

ensity. The specific, you, and, and the user.

.


 and.

.






















, and, and, and the user, and, and.

with the user, and, and, and, and, the data, and the system, you, and, andling, and the user, and the data.
. The following.

.

2, and to the, and, and, and, and, and, the system, and, the data, and, and, for the system, and, the system, the user, the, the system, and, the system, and, the system.


s, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, the assistant, the

,

, specifically, the user, the
20,

, and, the
, the data, the specific, the user,

, the user, and/
, the specific, the user, the














 20, the



3,

2,



2,
24,
3,



 20
32,



 2, 2
 2od
 2, 2, 2, 2, 2, 2
 2, you, you, 2, 2, there, there, there, 2, 2,
22, the user, and, 2,
,


 2,
 2, the 2, and, the
2,
 2, which, the
3an,

, there, there, and, and,

, and, there.

?RESS, there.



q, there.



2, there, but, but


, the user,


 the

 the user, there, there, there, there, there, there, there, you, there, there, there, there, there, there, there, there. There. There. There. The following, a, which, there, there,
2, and
 20,
, which, there, there, there, there, a, the

,

, the
 of
  0, and IT,



, the
2,


ors.
 0,
 RESS, the

 of
ing, which of the user, which, users,
 of
,

 of the of the user, of
. of  of the
, the
,
of the 1, a user, 0,


ilater.0,
0, the user of the user, of the user, one of the 0, a user, a, the user of the user.
.
2, the user of the user of  of

 of the following of the user, the user of the 8y of the user, the user.RESS.s. 8, the  the first, the of the user, which, of the 8 of the user. The 0, the user of the user. The first, which, the
  the of the data. The re, the user, which of the user, which of the, which of the following, the of the
 of the of the of the of the next, of the first, the first, the of the " of the next of the mains. of the "it of the user of the of the of the user of the for the, which, of the of the user, where, of the user, which of the user,
 the user of the user of the user, of the of the user, of the user of the user of the user of the user of the user, of the user of the user of the user, of the user of the user of the user of the user, a user.
ats, 0 of the user, of anatom, which of the user, which, which of the main, of the of which of which of which of the sample of the sample of the user. of the sample of the device, of the sample of the, of the user, of the sample, of the sample,

3, 8, of the sample of the of 4 of 20iled of
ofing, of one, 2, 4 of the one. 2, 2, 4iting.ing. of  of  of  of il of the, which of the, of one of one, which of the, of the, which of the "m, which of
ofic of the, for the of the of the user of the user of 1, the 2, 2, 3,

, which of the
m, of the, of the of the of the device, of the device of 2, which, which, which of the of the of the of the user, which, which of which of which of which of which of which of which of which of the of the of the for the level, which is of
, of the same, which of the sample, which is a sample, which of the sample, which, which, which is a specific of the of which of is of
m, which of the of which of which is for which for which for which for
for, which, which, which,

,

, 8, of the for
of, 1, which, which for the, for the

is, for

im, and m3, which of
0im, which is the
ofilits, 3, which, whiches, which, which, which

The behavior is always the following: before outputting this pile of garbage, the application freezes for a few seconds after I send the prompt, then this comes out.

[leestott]

@natke did you repo this issue?

[natke]

I have reproduced the issue with the 4k CPU model.

I did not reproduce it with the 128k CPU model. This is the output I get for the 128k CPU:

Output:  Certainly! Ireland, officially known as the Republic of Ireland, is an island nation in Northwestern Europe. It is bordered by the Atlantic Ocean to the west and the Irish Sea to the east. The Republic of Ireland covers about five-eighths of the island, while Northern Ireland, a part of the United Kingdom, occupies the remaining three-eighths. The capital city of Ireland is Dublin, which is also the largest city in the country.\n\nIreland has a rich history that dates back thousands of years. The earliest known settlers in Ireland were the Mesolithic hunter-gatherers, who arrived around 8000 BC. The country was later inhabited by the Neolithic farmers, who introduced agriculture and settled life. The Iron Age saw the arrival of the Celts, who brought with them their language, culture, and mythology.\n\nIreland's history is marked by periods of conquest, colonization, and rebellion. The country was invaded by the Romans, Vikings, and Normans, each leaving their mark on the landscape and culture. The most significant period of colonization came in the 16th and 17th centuries, when the English and Scottish crowns sought to bring Ireland under their control. This led to the plantation of Ulster, the suppression of the native Irish language, and the persecution of the Catholic majority.\n\nThe struggle for Irish independence from British rule culminated in the Easter Rising of 1916, a rebellion that aimed to establish an independent Irish Republic. The uprising was suppressed, but it galvanized support for the cause of Irish nationalism. The subsequent War of Independence, which lasted from 1919 to 1921, resulted in the partition of Ireland and the establishment of the Irish Free State, a dominion of the British Commonwealth.\n\nIn 1949, the Irish Free State became a fully independent republic, and the country became known as the Republic of Ireland. Northern Ireland, however, remained part of the United Kingdom, and the island of Ireland continued to be a source of political tension and conflict, particularly in the form of the Troubles, a period of sectarian violence that lasted from the late 1960s to the late 1990s.\n\nIreland is a parliamentary democracy with a president as head of state and a prime minister as head of government. The country is a member of the European Union (EU), and the Irish language, or Gaeilge, is one of the EU's 24 official languages.\n\nIreland is known for its natural beauty, with lush green landscapes, rolling hills, and rugged coastlines. The country is also a major producer of dairy products, beef, and potatoes. The economy of Ireland is one of the most open and competitive in the world, with a strong emphasis on technology, finance, and tourism.\n\nIreland has a rich cultural heritage, with a history of music, literature, and art that dates back centuries. Traditional Irish music, with its lively jigs, reels, and ballads, remains an important part of the country's cultural identity. The country is also known for its literature, with famous authors such as James Joyce, W.B. Yeats, and Samuel Beckett. Irish art, particularly painting and sculpture, has also made significant contributions to the world of art.\n\nIreland is a popular tourist destination, with many visitors coming to see its stunning landscapes, historic sites, and vibrant cities. The country is also known for its festivals, such as the traditional St. Patrick's Day celebrations in Dublin and the spectacular Galway Fleadh, a festival of music and dance.\n\nIn summary, Ireland is a country with a rich history, vibrant culture, and beautiful landscapes. It is a place of great natural beauty, with a strong sense of community and a deep connection to its past. The country continues to evolve, embracing its heritage while looking to the future.

I will try with the 128k DirectML model

[natke]

We are investigating this, as part of this issue: microsoft/onnxruntime-genai#552

[kinfey]

It is recommended that you use ONNX Runtime for GenAI to call Phi-3 to obtain the best results.

[myadav2]

I've been experiencing a similar issue when fine-tuning phi3 and phi3.5 models as well. Produces a lot of tokens at the end that are gibberish even after fine-tuning.

Looks like this has been solved with the latest ONNX release, but fine-tuning these ONNX models by converting them to torch is really tricky. Any solution for that?

[zedhaque]

Just confirming it's still happening on all Phi-3 family models. I have tested it, though I have yet to try 3.5. Interestingly, it occurs when we exceed a 4K context window.

[LoriBru]

I also confirm it is still happening with the latest ONNX release and Phi-3 models.

[leestott]

Closing issue as no issues have been reported since Oct 2024 if you experience issues please reopen