Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add model ids corresponding to inference profiles supported by AWS Bedrock #5696

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

shrikrishnaholla
Copy link

AWS has something called "Inference Profiles" for performing cross-region inference for calling Bedrock APIs for specific models from specific regions. This is typically used by using particular model ids, which is added with this commit.

Read more here:

I noticed this when setting up Langflow and trying to use the models in Bedrock. It is a simple constants change for now. But maybe is it possible to allow a user to type-in a model id, if in the dropdown the option user tries to choose does not exist? Because API providers would keep adding new models every other week and we would need to keep playing catch up :)

…drock

AWS has something called "Inference Profiles" for performing cross-region inference for calling Bedrock APIs for specific models from specific regions. This is typically used by using particular model ids, which is added with this commit.

Read more here:
- https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html
- https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles-support.html
…-profiles-supported-models

Add model ids corresponding to inference profiles supported by AWS Bedrock
@dosubot dosubot bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Jan 15, 2025
Copy link

codspeed-hq bot commented Jan 15, 2025

CodSpeed Performance Report

Merging #5696 will degrade performances by 32.42%

Comparing taghash:main (1879eeb) with main (c7f6554)

Summary

⚡ 1 improvements
❌ 1 regressions
✅ 7 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark main taghash:main Change
test_successful_run_with_output_type_any 4,090.6 ms 246.6 ms ×17
test_successful_run_with_output_type_debug 167 ms 247.1 ms -32.42%

Copy link
Contributor

@ogabrielluiz ogabrielluiz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @shrikrishnaholla

What if we added the field to pick a region insteaD?

@shrikrishnaholla
Copy link
Author

@ogabrielluiz If you see in the linked docs, according to region + model, the model id should be changed. There doesn't seem to be a consistent logic there. It's just a table with values that need to be mapped. Hence I tried to keep it simple, since I'm blocked on this 😅

@shrikrishnaholla
Copy link
Author

How should we proceed with this? Please advise

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size:M This PR changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants