[TextGeneration] Add Streaming Functionality #1246

dsikka · 2023-09-14T22:03:20Z

Summary:

Enable the ability to stream the generated tokens as they're produced, as opposed to waiting for the completed generated sequence at the end.
Adds streaming as a boolean flag as an input which then toggles the class streaming flag
Each generated token when streaming is enabled, is its own instance of GeneratedText which also includes whether or not the stream is complete with the finish flag
Note: this makes the output from this pipeline a generator as well

Testing:

Locally Testing the following pipeline runs, with and without streaming:

Without streaming:

from deepsparse import Pipeline

pipeline = Pipeline.create(
   task="text_generation",
   model_path="/home/dsikka/.cache/sparsezoo/neuralmagic/opt-1.3b-opt_pretrain-quantW8A8/deployment",
   engine_type="onnxruntime"
)

inference = pipeline(sequences=["hello?", "cool"], num_generated_predictions=1, max_tokens=100)
for generation in inference:
   print(generation)

Output:

created=datetime.datetime(2023, 9, 18, 15, 33, 38, 804176) prompts=['hello?', 'cool'] generations=[GeneratedText(text='\n\nI am a new member of the forum and I am looking for some help. I am a new member of the forum and I am looking for some help. I am a new member of the forum and I am looking for some help. I am a new member of the forum and I am looking for some help. I am a new member of the forum and I am looking for some help. I am a new member of the forum and I am looking for some help. I am', score=None, finished=True, finished_reason='length'), GeneratedText(text=", i'll be there.\nI'll be there too.", score=None, finished=True, finished_reason='stop')] session_id=None

With streaming:

from deepsparse import Pipeline

pipeline = Pipeline.create(
   task="text_generation",
   model_path="/home/dsikka/.cache/sparsezoo/neuralmagic/opt-1.3b-opt_pretrain-quantW8A8/deployment",
   engine_type="onnxruntime"
)

inference = pipeline(sequences=["hello?", "cool"], num_generated_predictions=1, max_tokens=100, streaming=True)
for generation in inference:
   print(generation)

Output:

created=datetime.datetime(2023, 9, 18, 15, 31, 48, 948891) prompts=['hello?', 'cool'] generations=[GeneratedText(text='\n', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 49, 155116) prompts=['hello?', 'cool'] generations=[GeneratedText(text='I', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 49, 325024) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' am', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 49, 530060) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' a', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 49, 689869) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' new', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 49, 846707) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' member', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 50, 6572) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' of', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 50, 166338) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' the', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 50, 324865) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' forum', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 50, 483855) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' and', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 50, 642005) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' I', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 50, 799761) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' am', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 50, 957431) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' looking', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 51, 118104) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' for', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 51, 278309) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' some', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 52, 364368) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' help', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 53, 133188) prompts=['hello?', 'cool'] generations=[GeneratedText(text='.', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 53, 307915) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' I', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 53, 521983) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' am', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 54, 368003) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' a', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 54, 504268) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' new', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 54, 652974) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' member', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 54, 801083) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' of', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 54, 938365) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' the', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 55, 88040) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' forum', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 55, 235587) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' and', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 55, 372666) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' I', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 55, 521513) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' am', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 56, 268422) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' looking', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 57, 724933) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' for', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 57, 911957) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' some', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 58, 35650) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' help', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 58, 179187) prompts=['hello?', 'cool'] generations=[GeneratedText(text='.', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 58, 314345) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' I', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 58, 450799) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' am', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 58, 587709) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' a', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 58, 710923) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' new', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 58, 846919) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' member', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 58, 988818) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' of', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 59, 156708) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' the', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 59, 299367) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' forum', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 59, 491682) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' and', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 59, 625725) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' I', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 59, 757834) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' am', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 31, 59, 918687) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' looking', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 0, 70859) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' for', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 0, 248243) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' some', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 0, 381889) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' help', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 0, 513949) prompts=['hello?', 'cool'] generations=[GeneratedText(text='.', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 0, 636205) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' I', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 0, 769554) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' am', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 0, 904147) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' a', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 1, 39724) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' new', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 1, 171689) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' member', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 1, 320092) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' of', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 1, 721389) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' the', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 1, 872003) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' forum', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 2, 9602) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' and', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 2, 145309) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' I', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 2, 280406) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' am', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 2, 417418) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' looking', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 2, 559980) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' for', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 2, 696098) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' some', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 2, 828610) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' help', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 2, 964153) prompts=['hello?', 'cool'] generations=[GeneratedText(text='.', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 3, 141259) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' I', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 3, 277610) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' am', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 3, 409814) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' a', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 3, 548530) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' new', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 3, 715077) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' member', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 3, 850432) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' of', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 3, 975822) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' the', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 4, 108335) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' forum', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 4, 241856) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' and', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 4, 367432) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' I', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 4, 499662) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' am', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 4, 635489) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' looking', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 4, 770197) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' for', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 4, 927679) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' some', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 5, 99002) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' help', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 5, 242596) prompts=['hello?', 'cool'] generations=[GeneratedText(text='.', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 5, 418254) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' I', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 5, 552523) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' am', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 5, 687031) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' a', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 5, 822692) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' new', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 5, 954573) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' member', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 6, 93444) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' of', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 6, 243410) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' the', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 6, 377820) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' forum', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 6, 513661) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' and', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 6, 649768) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' I', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 6, 785508) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' am', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 6, 925207) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' looking', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 7, 60518) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' for', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 7, 195431) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' some', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 7, 330862) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' help', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 7, 465498) prompts=['hello?', 'cool'] generations=[GeneratedText(text='.', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 7, 601641) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' I', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 7, 734736) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' am', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 7, 734947) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' am', score=None, finished=True, finished_reason='length')] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 8, 209591) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' i', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 8, 382645) prompts=['hello?', 'cool'] generations=[GeneratedText(text="'ll", score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 8, 524858) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' be', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 8, 700586) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' there', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 8, 832786) prompts=['hello?', 'cool'] generations=[GeneratedText(text='.', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 8, 956063) prompts=['hello?', 'cool'] generations=[GeneratedText(text='\n', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 9, 99501) prompts=['hello?', 'cool'] generations=[GeneratedText(text='I', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 9, 279348) prompts=['hello?', 'cool'] generations=[GeneratedText(text="'ll", score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 9, 423291) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' be', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 9, 599410) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' there', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 9, 741689) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' too', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 9, 921228) prompts=['hello?', 'cool'] generations=[GeneratedText(text='.', score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 10, 65757) prompts=['hello?', 'cool'] generations=[GeneratedText(text='</s>', score=None, finished=True, finished_reason='stop')] session_id=None

src/deepsparse/transformers/pipelines/text_generation.py

tests/deepsparse/transformers/pipelines/test_text_generation.py

src/deepsparse/transformers/pipelines/text_generation.py

tests/deepsparse/transformers/pipelines/test_text_generation.py

dbogunowicz · 2023-09-21T13:32:22Z

Something maybe parallel to @mgoin mentions, one thing that I find a bit problematic is this:

created=datetime.datetime(2023, 9, 18, 15, 32, 9, 279348) prompts=['hello?', 'cool'] generations=[GeneratedText(text="'ll", score=None, finished=False, finished_reason=None)] session_id=None
created=datetime.datetime(2023, 9, 18, 15, 32, 9, 423291) prompts=['hello?', 'cool'] generations=[GeneratedText(text=' be', score=None, finished=False, finished_reason=None)] session_id=None

The "lll" and "be" tokens, we do not know whether they came from "hello?" or "cool" prompt. For now, the user does not know the relationship between the generated token and the original prompt (which egg was hatched by which chicken in plain english :)). Clearly, it's not straightforward to resolve if there is no unique identifier like session_ids to convey this information...

bfineran

LGTM - let's check in some tests for this as well

…utput

src/deepsparse/pipeline.py

src/deepsparse/transformers/pipelines/text_generation.py

src/deepsparse/pipeline.py

Co-authored-by: Rahul Tuli <[email protected]>

dsikka force-pushed the textgen_update branch from 115a7d7 to 6a79bca Compare September 15, 2023 17:39

dsikka marked this pull request as ready for review September 17, 2023 19:57

dsikka force-pushed the enable_streaming branch from b610646 to a69abf9 Compare September 18, 2023 16:13

dsikka requested review from dbogunowicz, bfineran, Satrat, mgoin and rahul-tuli September 18, 2023 19:35

Base automatically changed from textgen_update to main September 19, 2023 10:04

dsikka force-pushed the enable_streaming branch from 627b849 to ce85c4b Compare September 19, 2023 14:26

bfineran reviewed Sep 19, 2023

View reviewed changes

src/deepsparse/transformers/pipelines/text_generation.py Outdated Show resolved Hide resolved

src/deepsparse/transformers/pipelines/text_generation.py Outdated Show resolved Hide resolved

tests/deepsparse/transformers/pipelines/test_text_generation.py Outdated Show resolved Hide resolved

mgoin reviewed Sep 20, 2023

View reviewed changes

src/deepsparse/transformers/pipelines/text_generation.py Outdated Show resolved Hide resolved

dbogunowicz previously approved these changes Sep 21, 2023

View reviewed changes

tests/deepsparse/transformers/pipelines/test_text_generation.py Show resolved Hide resolved

dsikka dismissed dbogunowicz’s stale review via 1dc2bd1 September 21, 2023 17:31

dsikka requested review from bfineran, dbogunowicz and mgoin September 21, 2023 17:33

bfineran previously approved these changes Sep 21, 2023

View reviewed changes

Satrat previously approved these changes Sep 21, 2023

View reviewed changes

dsikka added 10 commits September 21, 2023 14:18

add streaming functionality

46361d2

remove print

c0c7e37

set back default value

ab838e7

rebase

7c2ec5d

update to yield

444f0a2

update pipeline.py

5bd29aa

update tests

20a46a7

refactor out streaming functions and remove yield in process_engine_o…

fb38e2f

…utput

fix tests

fd40bf9

update pipeline to use kwargs

592c81c

rebase

3a18e5f

dsikka dismissed stale reviews from Satrat and bfineran via 3a18e5f September 21, 2023 18:29

dsikka force-pushed the enable_streaming branch from 1dc2bd1 to 3a18e5f Compare September 21, 2023 18:29

Satrat self-requested a review September 21, 2023 18:31

Satrat previously approved these changes Sep 21, 2023

View reviewed changes

rahul-tuli reviewed Sep 21, 2023

View reviewed changes

mgoin reviewed Sep 21, 2023

View reviewed changes

src/deepsparse/pipeline.py Show resolved Hide resolved

Update src/deepsparse/transformers/pipelines/text_generation.py

b833a22

Co-authored-by: Rahul Tuli <[email protected]>

dsikka dismissed Satrat’s stale review via b833a22 September 21, 2023 19:43

Update src/deepsparse/pipeline.py

7db3f39

Co-authored-by: Rahul Tuli <[email protected]>

bfineran approved these changes Sep 21, 2023

View reviewed changes

bfineran merged commit cd74aa2 into main Sep 21, 2023

bfineran deleted the enable_streaming branch September 21, 2023 20:11

rahul-tuli mentioned this pull request Sep 21, 2023

Add minor updates #1265

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TextGeneration] Add Streaming Functionality #1246

[TextGeneration] Add Streaming Functionality #1246

dsikka commented Sep 14, 2023 •

edited by mgoin

Loading

dbogunowicz commented Sep 21, 2023 •

edited

Loading

bfineran left a comment

[TextGeneration] Add Streaming Functionality #1246

[TextGeneration] Add Streaming Functionality #1246

Conversation

dsikka commented Sep 14, 2023 • edited by mgoin Loading

Summary:

Testing:

dbogunowicz commented Sep 21, 2023 • edited Loading

bfineran left a comment

Choose a reason for hiding this comment

dsikka commented Sep 14, 2023 •

edited by mgoin

Loading

dbogunowicz commented Sep 21, 2023 •

edited

Loading