Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: adds type=CEL LLM cost in Test_Examples_TokenRateLimit #257

Open
mathetake opened this issue Jan 31, 2025 · 0 comments
Open

test: adds type=CEL LLM cost in Test_Examples_TokenRateLimit #257

mathetake opened this issue Jan 31, 2025 · 0 comments
Assignees
Labels
enhancement New feature or request test

Comments

@mathetake
Copy link
Member

Description:

Currently, examples/token_rate_limit has only input/output/total token costs configurations, but there's another option that can be used which is CEL expression:

// CELExpression is the CEL expression to calculate the cost of the request.
// The CEL expression must return a signed or unsigned integer. If the
// return value is negative, it will be error.
//
// The expression can use the following variables:
//
// * model: the model name extracted from the request content. Type: string.
// * backend: the backend name in the form of "name.namespace". Type: string.
// * input_tokens: the number of input tokens. Type: unsigned integer.
// * output_tokens: the number of output tokens. Type: unsigned integer.
// * total_tokens: the total number of tokens. Type: unsigned integer.
//
// For example, the following expressions are valid:
//
// * "model == 'llama' ? input_tokens + output_token * 0.5 : total_tokens"
// * "backend == 'foo.default' ? input_tokens + output_tokens : total_tokens"
// * "input_tokens + output_tokens + total_tokens"
// * "input_tokens * output_tokens"
//
// +optional
CELExpression *string `json:"celExpression,omitempty"`

I would like to showcase the use of CEL expression cost API and add test in
https://github.com/envoyproxy/ai-gateway/blob/main/tests/e2e/token_ratelimit_test.go

@mathetake mathetake added enhancement New feature or request good first issue Good for newcomers labels Jan 31, 2025
@mathetake mathetake added test and removed good first issue Good for newcomers labels Jan 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request test
Projects
None yet
Development

No branches or pull requests

2 participants