New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature: Bedrock converse integration #678

Merged

VisargD merged 8 commits into Portkey-AI:main from narengogi:feat/converse-api

Oct 23, 2024

Collaborator

narengogi commented Oct 14, 2024 •

edited

Loading

Title:

This replaces the need to have a tokenizer for open source models hosted on AWS bedrock, converse is bedrock's unified gateway like implementation

Limitations:

Few legacy models are not supported in the converse endpoint (https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-supported-models-features.html)

Testing Done:

Verified stream chunk handing of older routes
Tool calling and Tool Responses in chat messages, single and multi turn
Chat Completions
Tested with different models
Tested prompting with images

Related Issues: (optional)

[Feature] Create tokenizer for applying chat template for mistral, llama etc #625

narengogi force-pushed the feat/converse-api branch from bc10b16 to 38f59fb Compare

October 14, 2024 11:18

VisargD reviewed

View reviewed changes

src/providers/bedrock/index.ts Outdated Show resolved Hide resolved

VisargD reviewed

View reviewed changes

src/providers/bedrock/chatComplete.ts Outdated Show resolved Hide resolved

VisargD reviewed

View reviewed changes

src/providers/bedrock/chatComplete.ts Outdated Show resolved Hide resolved

VisargD reviewed

View reviewed changes

src/providers/bedrock/chatComplete.ts Show resolved Hide resolved

VisargD reviewed

View reviewed changes

src/providers/bedrock/index.ts Outdated Show resolved Hide resolved

VisargD reviewed

View reviewed changes

src/providers/bedrock/chatComplete.ts Outdated Show resolved Hide resolved

narengogi requested a review from VisargD

October 15, 2024 14:52

VisargD reviewed

View reviewed changes

src/providers/bedrock/index.ts Outdated Show resolved Hide resolved

VisargD reviewed

View reviewed changes

src/providers/bedrock/chatComplete.ts Outdated Show resolved Hide resolved

narengogi force-pushed the feat/converse-api branch 2 times, most recently from 1a96228 to 84b2a6b Compare

October 22, 2024 15:24


          Bedrock converse integration

7ca34d4

fix tool choice

fix tool choice

fix interface

add support for prompting with images

changes per comments

changes per comments

Add back support for Cohere command and AI21 Jurrasic 2 models because they are not supported by bedrock converse API

handle prompting with documents on bedrock converse

support prompting with documents

narengogi requested a review from VisargD

October 22, 2024 15:28

narengogi force-pushed the feat/converse-api branch from 84b2a6b to 7ca34d4 Compare

October 22, 2024 15:28


          add comments

33f1bfa

VisargD reviewed

View reviewed changes

src/providers/bedrock/chatComplete.ts Show resolved Hide resolved


          add support for max_completion_tokens

913a414

VisargD reviewed

View reviewed changes

src/providers/bedrock/utils.ts Show resolved Hide resolved

narengogi added 4 commits

October 22, 2024 23:00


          Restrict known additional model fields to the specific providers

d5e797e


          Fix interfaces

f13d1cf


          fix key error

878ad9a


          make function optional

b0e32c9

VisargD approved these changes

View reviewed changes


          Merge branch 'main' into feat/converse-api

862ae96

VisargD merged commit 3c75441 into Portkey-AI:main

1 check passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet