Anthropic streaming support #652
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
piercefreeman
commented
Jan 9, 2025
tests/models/test_anthropic.py
Outdated
| class MockAsyncStream(AsyncStream[T]): | ||
| """Mock implementation of AsyncStream for testing.""" | ||
|
|
||
| def __init__(self, events: list[list[T]]): |
There was a problem hiding this comment.
To fulfill the contract of AsyncAnthropic, we need to pass create_mock a single instance of our MockAsyncStream. However since this async stream is recycled across a potentially multi-turn conversation with tool calls, we also need the ability to provide multiple chained sequences together.
We modify the existing MockAsyncStream class here to support list[list[Messages]] instead of a single event stream. If desired this change could be refactored into the original class.
dmontagu
force-pushed
the
dmontagu/refactor-streaming
branch
from
ae69e4e to
ed42efb
Compare
|
Thanks! Excited to review this. We're working on getting streaming across the line here soon. |
piercefreeman
force-pushed
the
feature/anthropic-streaming
branch
from
9e3fa70 to
256a3d9
Compare
|
@sydney-runkle Rebased off of @dmontagu's latest changes and have tests passing here. Should be ready for a first review. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Branching off @dmontagu's pending streaming refactor in #468. I'll rebase this off of main once that work is merged in.
This PR adds support for streaming responses from Anthropic's API. This is a pretty straightforward implementation inline with OpenAI and Grok handlers, with the exception of having to locally assemble the json arguments to functions progressively - since Claude architectures can yield function inputs across multiple messages. The JSON is not actually valid until the last delta message.