Phi-3-mini-4k-instruct-onnx model generates nonsensical results when prompt is longer than half of the context window length

[jackylu0124]

I am running the Phi-3-mini-4k-instruct-onnx model on desktop CPU, and one behavior I have noticed is that, after the back and forth conversation is longer than half of the context window length (in other words, longer than 2048 tokens), the model starts generating nonsensical and unreadable results. This is weird because I would expect the model to continue generating readable results at least to the point that's closer to the end of the context window length. I would really appreciate any insights on this problematic behavior as well as any ways that I can fix this issue. Thanks a lot in advance!

To reproduce this issue, you can run the example phi3-qa.py script (https://github.com/microsoft/onnxruntime-genai/blob/main/examples/python/phi3-qa.py) (with command like python phi3-qa.py -m .\Phi-3-mini-4k-instruct-onnx\cpu_and_mobile\cpu-int4-rtn-block-32-acc-level-4) with the if-block setting search_options['max_length'] = 2048 commented out to allow longer input and also replace the prompt in the line input_tokens = tokenizer.encode(prompt)(

onnxruntime-genai/examples/python/phi3-qa.py

Line 38 in 6be8835

input_tokens = tokenizer.encode(prompt)

) with the following conversation string (which is a conversation between the user and the assistant, and the string corresponds to 2271 tokens):

<|user|>\nTell me about Paris, France.<|end|>\n<|assistant|>\nParis, the capital city of France, is renowned for its rich history, iconic landmarks, and vibrant culture. Known as "The City of Light," Paris is situated in the north-central part of the country along the Seine River.\n\nHere are some key aspects of Paris:\n\n1. Landmarks: Paris is home to numerous famous landmarks, including the Eiffel Tower, the Louvre Museum, Notre-Dame Cathedral, and the Champs-Élysées. The Eiffel Tower, built in 1889, is an iconic symbol of Paris and attracts millions of tourists each year. The Louvre Museum, the world\'s largest art museum, houses thousands of works of art, including the Mona Lisa and the Venus de Milo.\n\n2. History: Paris has a rich history dating back to the 3rd century BC, when it was founded by a Celtic tribe called the Parisii. Over the centuries, the city has been influenced by various cultures, including the Romans, the Franks, and the Normans. The French Revolution in the late 18th century marked a significant turning point in Paris\'s history, leading to the establishment of the modern French Republic.\n\n3. Culture: Paris is a global center for art, fashion, gastronomy, and culture. The city is home to numerous museums, including the Centre Pompidou, Musée d\'Orsay, and Musée Rodin. Paris is also known for its fashion industry, with many famous designers having their origins in the city. The city\'s cuisine is also highly regarded, with a focus on fresh ingredients, and a wide variety of dishes, including French classics like coq au vin, boeuf bourguignon, and crêpes.\n\n4. Architecture: Parisian architecture is characterized by its diverse styles, ranging from Gothic and Romanesque to Art Nouveau and Art Deco. The city\'s famous Haussmannian buildings, designed by Baron Haussmann in the mid-19th century, are known for their uniform facades, wrought-iron balconies, and large windows.\n\n5. Transportation: Paris has an extensive public transportation system, including the Paris Métro, RER (suburban trains), and buses. The city\'s iconic yellow taxis are also a popular mode of transportation.\n\n6. Language: The official language of Paris is French, and the city\'s residents are known for their charm and politeness.\n\n7. Festivals and Events: Paris hosts numerous festivals and events throughout the year, including the annual Bastille Day celebrations, the Paris Fashion Week, and the famous annual New Year\'s Eve fireworks on the Eiffel Tower.\n\n8. Geography: Paris is located in the north-central part of France, with the Seine River running through the city. The city\'s geography is characterized by rolling hills and picturesque parks, such as the Bois de Boulogne and the Jardin des Tuileries.\n\n9. Population: As of 2021, Paris has an estimated population of around 2.2 million residents, with the metropolitan area housing over 12 million people.\n\nIn summary, Paris is a city steeped in history, culture, and art, with a unique blend of architectural styles and a vibrant atmosphere that continues to captivate millions of visitors each year.<|end|>\n<|user|>\nPlease give me a list of 5 architectural landmarks in Paris, France.<|end|>\n<|assistant|>\n1. Eiffel Tower: Designed by Gustave Eiffel and completed in 1889, the Eiffel Tower is an iconic symbol of Paris and France. Standing at 324 meters tall, it was the tallest man-made structure in the world until the completion of the Chrysler Building in New York in 1930. The Eiffel Tower is made of wrought iron and offers visitors stunning views of the city from its three levels.\n\n2. Notre-Dame Cathedral: Located on the Île de la Cité, Notre-Dame Cathedral is a masterpiece of French Gothic architecture. Construction began in the 12th century and continued for over 200 years, with the cathedral\'s completion in the 14th century. The cathedral is famous for its intricate facade, stained-glass windows, and the iconic gargoyles and chimeras.\n\n3. Louvre Museum: Originally built as a fortress in the 12th century, the Louvre Museum is now the world\'s largest art museum and a historic monument in Paris. The museum\'s most famous landmark is the iconic glass pyramid entrance, designed by architect I. M. Pei in the 1980s. The Louvre houses over 380,000 works of art, including the Mona Lisa and the Venus de Milo.\n\n4. Sacré-Cœur Basilica: The Sacré-Cœur Basilica, also known as the Basilique du Sacré-Cœur, is a Roman Catholic church and minor basilica located at the summit of the butte Montmartre, the highest point in Paris. The basilica was designed by Paul Abadie and dedicated in 1914. Its white domes and lavender-colored travertine stone make it a distinctive landmark in the Paris skyline.\n\n5. Arc de Triomphe: The Arc de Triomphe is a monumental structure located at the western end of the Champs-Élysées. Commissioned by Napoleon in 1806, the Arc was designed by Jean-François-Thérèse Chalgrin and completed in 1836. The monument honors those who fought and died for France during the French Revolutionary and Napoleonic Wars. The Arc features sculptural reliefs and inscriptions, and its façade is adorned with the names of 357 generals and 660 soldiers.\n\nThese five architectural landmarks showcase the diverse styles and historical periods of Paris, from Gothic to Neoclassical, and from the 19th to the 20th centuries. Each landmark has its unique features and contributes to the city\'s rich architectural heritage.<|end|>\n<|user|>\nPlease give me a list of 10 famous items displayed in the Louvre Museum. Thanks!<|end|>\n<|assistant|>\n1. Mona Lisa: The Mona Lisa, painted by Leonardo da Vinci in the early 16th century, is arguably the most famous painting in the world. The portrait is known for its enigmatic smile and masterful use of sfumato, a technique that creates a soft, hazy effect.\n\n2. Venus de Milo: This ancient Greek statue, believed to have been created around 130-100 BC, is a masterpiece of Hellenistic sculpture. The Venus de Milo is renowned for its graceful beauty and the mystery surrounding its missing arms.\n\n3. Winged Victory of Samothrace: This Hellenistic sculpture, dating back to the 2nd century BC, depicts the Greek goddess Nike, the personification of victory. The sculpture is celebrated for its dynamic movement and intricate details.\n\n4. Liberty Leading the People: This iconic painting by Eugène Delacroix, created in 1830, commemorates the July Revolution in France. The artwork depicts a woman personifying Liberty leading a group of revolutionaries over the bodies of the fallen.\n\n5. The Wedding at Cana: A 1516 painting by Veronese, The Wedding at Cana is a large-scale work that depicts the biblical story of Jesus turning water into wine at a wedding feast. The painting is known for its vibrant colors and intricate details.\n\n6. The Raft of the Medusa: This 1819 painting by Théodore Géricault is a powerful depiction of the aftermath of the shipwreck of the French frigate Méduse. The painting is famous for its dramatic composition and emotional intensity.\n\n7. The Coronation of Napoleon: This 1805 painting by Jacques-Louis David portrays the coronation of Napoleon Bonaparte as Emperor of the French. The artwork is a masterpiece of neoclassical style and captures the pomp and ceremony of the event.\n\n8. The Death of Socrates: A 1914 painting by Pablo Picasso, The Death of Socrates depicts the moment when the ancient Greek philosopher Socrates drinks the poison hemlock, as ordered by the Athenian government. The painting is a powerful expression of the existential themes associated with Socrates\' death.\n\n9. The Turkish Bath: A 1887 painting by Puvis de Chavannes, The Turkish Bath is a large-scale work that depicts a scene of relaxation and leisity in a traditional Turkish bath. The painting is known for its subdued color palette and serene atmosphere.\n\n10. The Wedding at Cana: A 1901 painting by Giovanni Boldini, The Wedding at Cana is a lively depiction of a festive Italian wedding. The painting is characterized by Boldini\'s signature flamboyant style and his ability to capture the essence of the subjects with a sense of movement and energy.\n\nThese 10 famous items in the Louvre Museum showcase a diverse range of artistic styles and periods, from ancient Greek and Roman sculptures to Renaissance and 19th-century paintings. The collection at the Louvre is one of the world\'s largest and most important museums, housing over 800,000 works of art, and these famous items are just a small selection of the museum\'s vast collection.<|end|>\n<|user|>\nPlease give me a list of 10 famous French dishes.<|end|>\n<|assistant|>\n

For greater readability, the conversation string above corresponds the following dialog:

<|user|>
Tell me about Paris, France.<|end|>
<|assistant|>
Paris, the capital city of France, is renowned for its rich history, iconic landmarks, and vibrant culture. Known as "The City of Light," Paris is situated in the north-central part of the country along the Seine River.

Here are some key aspects of Paris:

1. Landmarks: Paris is home to numerous famous landmarks, including the Eiffel Tower, the Louvre Museum, Notre-Dame Cathedral, and the Champs-Élysées. The Eiffel Tower, built in 1889, is an iconic symbol of Paris and attracts millions of tourists each year. The Louvre Museum, the world's largest art museum, houses thousands of works of art, including the Mona Lisa and the Venus de Milo.

2. History: Paris has a rich history dating back to the 3rd century BC, when it was founded by a Celtic tribe called the Parisii. Over the centuries, the city has been influenced by various cultures, including the Romans, the Franks, and the Normans. The French Revolution in the late 18th century marked a significant turning point in Paris's history, leading to the establishment of the modern French Republic.

3. Culture: Paris is a global center for art, fashion, gastronomy, and culture. The city is home to numerous museums, including the Centre Pompidou, Musée d'Orsay, and Musée Rodin. Paris is also known for its fashion industry, with many famous designers having their origins in the city. The city's cuisine is also highly regarded, with a focus on fresh ingredients, and a wide variety of dishes, including French classics like coq au vin, boeuf bourguignon, and crêpes.

4. Architecture: Parisian architecture is characterized by its diverse styles, ranging from Gothic and Romanesque to Art Nouveau and Art Deco. The city's famous Haussmannian buildings, designed by Baron Haussmann in the mid-19th century, are known for their uniform facades, wrought-iron balconies, and large windows.

5. Transportation: Paris has an extensive public transportation system, including the Paris Métro, RER (suburban trains), and buses. The city's iconic yellow taxis are also a popular mode of transportation.

6. Language: The official language of Paris is French, and the city's residents are known for their charm and politeness.

7. Festivals and Events: Paris hosts numerous festivals and events throughout the year, including the annual Bastille Day celebrations, the Paris Fashion Week, and the famous annual New Year's Eve fireworks on the Eiffel Tower.

8. Geography: Paris is located in the north-central part of France, with the Seine River running through the city. The city's geography is characterized by rolling hills and picturesque parks, such as the Bois de Boulogne and the Jardin des Tuileries.

9. Population: As of 2021, Paris has an estimated population of around 2.2 million residents, with the metropolitan area housing over 12 million people.

In summary, Paris is a city steeped in history, culture, and art, with a unique blend of architectural styles and a vibrant atmosphere that continues to captivate millions of visitors each year.<|end|>
<|user|>
Please give me a list of 5 architectural landmarks in Paris, France.<|end|>
<|assistant|>
1. Eiffel Tower: Designed by Gustave Eiffel and completed in 1889, the Eiffel Tower is an iconic symbol of Paris and France. Standing at 324 meters tall, it was the tallest man-made structure in the world until the completion of the Chrysler Building in New York in 1930. The Eiffel Tower is made of wrought iron and offers visitors stunning views of the city from its three levels.

2. Notre-Dame Cathedral: Located on the Île de la Cité, Notre-Dame Cathedral is a masterpiece of French Gothic architecture. Construction began in the 12th century and continued for over 200 years, with the cathedral's completion in the 14th century. The cathedral is famous for its intricate facade, stained-glass windows, and the iconic gargoyles and chimeras.

3. Louvre Museum: Originally built as a fortress in the 12th century, the Louvre Museum is now the world's largest art museum and a historic monument in Paris. The museum's most famous landmark is the iconic glass pyramid entrance, designed by architect I. M. Pei in the 1980s. The Louvre houses over 380,000 works of art, including the Mona Lisa and the Venus de Milo.

4. Sacré-Cœur Basilica: The Sacré-Cœur Basilica, also known as the Basilique du Sacré-Cœur, is a Roman Catholic church and minor basilica located at the summit of the butte Montmartre, the highest point in Paris. The basilica was designed by Paul Abadie and dedicated in 1914. Its white domes and lavender-colored travertine stone make it a distinctive landmark in the Paris skyline.

5. Arc de Triomphe: The Arc de Triomphe is a monumental structure located at the western end of the Champs-Élysées. Commissioned by Napoleon in 1806, the Arc was designed by Jean-François-Thérèse Chalgrin and completed in 1836. The monument honors those who fought and died for France during the French Revolutionary and Napoleonic Wars. The Arc features sculptural reliefs and inscriptions, and its façade is adorned with the names of 357 generals and 660 soldiers.

These five architectural landmarks showcase the diverse styles and historical periods of Paris, from Gothic to Neoclassical, and from the 19th to the 20th centuries. Each landmark has its unique features and contributes to the city's rich architectural heritage.<|end|>
<|user|>
Please give me a list of 10 famous items displayed in the Louvre Museum. Thanks!<|end|>
<|assistant|>
1. Mona Lisa: The Mona Lisa, painted by Leonardo da Vinci in the early 16th century, is arguably the most famous painting in the world. The portrait is known for its enigmatic smile and masterful use of sfumato, a technique that creates a soft, hazy effect.

2. Venus de Milo: This ancient Greek statue, believed to have been created around 130-100 BC, is a masterpiece of Hellenistic sculpture. The Venus de Milo is renowned for its graceful beauty and the mystery surrounding its missing arms.

3. Winged Victory of Samothrace: This Hellenistic sculpture, dating back to the 2nd century BC, depicts the Greek goddess Nike, the personification of victory. The sculpture is celebrated for its dynamic movement and intricate details.

4. Liberty Leading the People: This iconic painting by Eugène Delacroix, created in 1830, commemorates the July Revolution in France. The artwork depicts a woman personifying Liberty leading a group of revolutionaries over the bodies of the fallen.

5. The Wedding at Cana: A 1516 painting by Veronese, The Wedding at Cana is a large-scale work that depicts the biblical story of Jesus turning water into wine at a wedding feast. The painting is known for its vibrant colors and intricate details.

6. The Raft of the Medusa: This 1819 painting by Théodore Géricault is a powerful depiction of the aftermath of the shipwreck of the French frigate Méduse. The painting is famous for its dramatic composition and emotional intensity.

7. The Coronation of Napoleon: This 1805 painting by Jacques-Louis David portrays the coronation of Napoleon Bonaparte as Emperor of the French. The artwork is a masterpiece of neoclassical style and captures the pomp and ceremony of the event.

8. The Death of Socrates: A 1914 painting by Pablo Picasso, The Death of Socrates depicts the moment when the ancient Greek philosopher Socrates drinks the poison hemlock, as ordered by the Athenian government. The painting is a powerful expression of the existential themes associated with Socrates' death.

9. The Turkish Bath: A 1887 painting by Puvis de Chavannes, The Turkish Bath is a large-scale work that depicts a scene of relaxation and leisity in a traditional Turkish bath. The painting is known for its subdued color palette and serene atmosphere.

10. The Wedding at Cana: A 1901 painting by Giovanni Boldini, The Wedding at Cana is a lively depiction of a festive Italian wedding. The painting is characterized by Boldini's signature flamboyant style and his ability to capture the essence of the subjects with a sense of movement and energy.

These 10 famous items in the Louvre Museum showcase a diverse range of artistic styles and periods, from ancient Greek and Roman sculptures to Renaissance and 19th-century paintings. The collection at the Louvre is one of the world's largest and most important museums, housing over 800,000 works of art, and these famous items are just a small selection of the museum's vast collection.<|end|>
<|user|>
Please give me a list of 10 famous French dishes.<|end|>
<|assistant|>

and then the model will generate the following nonsensical/unreadable output:

1. Coq au vin: Coq au vin is a classic French dish, traditionally made with chicken, and is a staple of French cuisine. It is a popular dish, known for its rich history and culture, and it is a culinary capital of Paris, France. It is a city, known for its history. The city, located in the world-class capital, known for its history, the city, is a city, located in France, known for its history, located in the city, known for its history, located in France, known for its history, located in France, known for its, known for its, known for its, the city, known for its, known for its, known for its, known for, known for its, known for, known for, known as, known for, known for France, and the city, known for its, and, the city, and its, known for its, known for the city, and its, and theater, the city, the E of France, France, the Seine the of the Eurbid, of France, theater. of France, and the Eats, known for its, and its, the city, and its, the Eat, the Etat, the city, and the Etat, of France, and itsre the, the Etat,tour, the Eustan the Eure, theater, aestain, the city, the Eent, the E the Eifface of the city, theat theater, the city, France, and itspanic-France, the first, itslift,savis, as the city,terrorent, as the city, the city, the city, the Seine, A, the city, the citytpear, French of France,-the city,tourngen theater of of theater,at, theater,, famous,lournourtourndenparis,sparis,theater,taunu8 by the by-by the by the of the of the city, France,and theater, of theater,theater,snapelle,-known-of theater,-theater,auustern, French by theater, French,latern8,-pourlater by theater,-Pin,-p leat rats,-when-France,theaterfam,-theater,later,  theaterlater, of the city,
- the the of the city-by theater,-
  theater,
-
-
-
-
- France, A,
  France, by theater,ust, by theat, by the by, theater, theater of France-the 2-b,ly, by the of the city-France,
- by theater,-the by,
  ,


  the   of


 -L
-Later-pam by by theater, , theater-L
 -B-by-by France, by-by-by this'
  by-bridesque of the of the city-p-p-p,-theaterre, ―grean-grean-pading,
-rom by quehroughts,- �squehridesqueadrough by the capital,88ree, by theater-when-obree, theater of the E-
- the original-the  of the of the of the 2-s of the
 by-to
 by the by by the  by  by by-by-de by 8 byck-by-of by 82-when 8
-the L by the by the by the   of
 of the  Paris-the  the
 each - 2-L-
 by  by by  France, by
-, ancient,ed-st-sthe-bron
-ston -  by of in       expresses,-De
-prets-
-au-Dezou by by by-De by:
-by by-by by
  by 4 by-by-by, by-brre-brre tockre by by  by-by-by-by the by by by the-fans by he-pano by  by-by-by-brouest8822 by
 of the
  of its of  - he au- ---of  D- 4-romorustime-backsou-Stre-Wusten-stime-Dout-Stor-comman-rome-
 -back,,,-the name-when-when France-inight,,, of pre-estimag 8agimag8 cusc, and and, Frenches of
 and-as by 0 by by the-F by by by utionicesagustuscune - utions -s of the ride by the by byime8uscults-f-F-and imeadimud of -F-F-F-the -F-Fundimeutimeursc-obredimeverultsimexagre by by bycult 8icesudise.-F-estandidesand8imevenutoures 8ime88ustouustustime8resultsparre,-   aille    you by theranustraning by the by by by by by by by-by-by by by by by estonundressesque-80on of statreversults by ultieuovertaunifond agwequequeunduscandshrime,- - on weau-estorn-over-estululustuscuston  of world-fouon by-deckar8ulultanking-deulullegulundouundulikeululululundundorulund by its byouundouonusts of offerter of 8888hrre-revingre88feringstandings by ofithivingivingvervingreheutreut-Deckututou044ragingultikehrustreicesesulting byronesque byckiceicesqueustime by byots bys by of resquequeufuscufs8880505iveidique85588uscuringing visitors of firmulike of including itsforestipeverice808orn-faillever by-widoorvingutstandingesqueuturingesparunearestromorvericeverorstandingesparesparenutods of0departmentututututuring of of of its of offering of itsime,-cobice, 2-0005hanferingice,-F-ffering8fering8ferferreneoveruturinguring-88quequequequequeque-F-F-overeoveril00ilifesteilimouses of offerde of by by by by by by by by of 8out008800 of thers of of its0il8000uting-estifestifestainest88est of of 880isisites of of itsitesque88estieu, the-festustedesque of cobearightourouououdequequequeque of of of Froning of its of itsque of of of of theque Aestestedesightesqueutorrenesque called namedile, andngenrenesque was itsndestronesqueun8000-follows0008ution by by the city by by by 0 byound80overvelorive of Parisite of 90000100est00re by by by 8outim8088880008888onites atl00000overite of80

[AMehlem]

I have the same issue. I get similar nonsensical output when using either the 4k-model or the 128k-model via the Onnx-Runtime with a long user prompt and setting the search option max_length to 3000. When using a shorter user prompt the output is as expected. Using the same user prompt on https://ai.azure.com/explore/models/Phi-3-mini-128k-
instruct/version/7/registry/azureml delivers correct results, so the model itself should not be the cause of the problem.

Tried with versions 0.2.0 and 0.3.0-rc2 of the library Microsoft.ML.OnnxRuntimeGenAI.DirectML.

P.S.: Thank you for this nice library, it is awesome to be able to run a SLM locally that easily.

[jackylu0124]

A quick follow-up to this issue. I would really appreciate any help or insights on this issue!

[bkaruman]

I am seeing a similar issue when running on CPU

[natke]

Hi @jackylu0124, @bkaruman, @AMehlem, I have reproduced your issue on CPU. We will investigate.

[jackylu0124]

Hi @natke, thank you for the update, I appreciate it.

[MaxAkbar]

I get the same issue if the conversation history goes above 2k (approximately) using .NET nugget package Microsoft.ML.OnnxRuntimeGenAI version 0.3.0. I am using the Phi-3-mini-4k-instruct-onnx. As a work around I truncate the conversation history (MaxTokenLength which is 4096 - 2500) and the issue goes away. I don't truncate the tokens but rather the conversation that the limit falls under.

EDIT: I can reproduce this with ollama version is 0.2.1 and it's Phi3 model phi3:mini. I get responses that are not entirely reflected in the conversation.

[baijumeswani]

This should be resolved with the PR: #802

[MaxAkbar]

Hello @baijumeswani, thank you for the update does this mean that we have to regenerate our onnx models? I also have used the model builder for the Phi-3.5 model.

[LukasKinder]

I was running experiments with the HotpotQA dataset. I also observe that the models performance drops considerably, if the number of tokens exceed ~2500.

(Y-Axis: Log-probability to generate the correct question, X-Axis: Number of Tokens in question)

[apsonawane]

Hello @MaxAkbar, I tested the latest onnxruntime-genai package with Phi-3.5 model, and it works fine when the input prompt is greater than 2k. Can you please try that?

[MaxAkbar]

Thank you @apsonawane , I also created an onnx model but will run some tests this weekend.

[myadav2]

I've been experiencing a similar issue when fine-tuning phi3 and phi3.5 models as well. Produces a lot of tokens at the end that are gibberish even after fine-tuning.

Looks like this has been solved with the latest ONNX release, but fine-tuning these ONNX models by converting them to torch is really tricky. Any solution for that?