Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Semantic conventions for LLM model server metrics #1102

Closed
achandrasekar opened this issue May 31, 2024 · 1 comment · Fixed by #1103
Closed

Semantic conventions for LLM model server metrics #1102

achandrasekar opened this issue May 31, 2024 · 1 comment · Fixed by #1103
Assignees
@AlexanderWert
Labels
area:gen-ai enhancement New feature or request experts needed This issue or pull request is outside an area where general approvers feel they can approve

Comments

@achandrasekar
Copy link
Contributor

Area(s)

area:gen-ai

Is your change request related to a problem? Please describe.

GenAI metrics today has client metrics. It would be good to add model server metrics to provide a standard set of performance and operational metrics for model servers. We have a proposal to standardize them as a part of Kubernetes Serving WG - https://docs.google.com/document/d/1SpSp1E6moa4HSrJnS4x3NpLuj88sMXr2tbofKlzTZpk/edit?resourcekey=0-ob5dR-AJxLQ5SvPlA4rdsg&tab=t.0#heading=h.qmzyorj64um1 (please request access if you are not able to access it). Having them standardized via LLM semantic conventions in open telemetry would be ideal.

Previous discussion on this in the LLM Semantic Conventions WG for context - https://docs.google.com/document/d/1EKIeDgBGXQPGehUigIRLwAUpRGa7-1kXB736EaYuJ2M/edit#bookmark=id.f4yu21sfndxj.

Describe the solution you'd like

Solution is to include these common model server metrics in the GenAI LLM semantic conventions

Describe alternatives you've considered

No response

Additional context

cc @lmolkova @SergeyKanzhelev
@achandrasekar achandrasekar added enhancement New feature or request experts needed This issue or pull request is outside an area where general approvers feel they can approve triage:needs-triage labels May 31, 2024
This was referenced May 31, 2024
Merged
Closed
@gyliu513
Copy link
Member

gyliu513 commented Jun 4, 2024 •
edited
Loading

Related with #1079

/cc @nirga

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AlexanderWert AlexanderWert

Labels
area:gen-ai enhancement New feature or request experts needed This issue or pull request is outside an area where general approvers feel they can approve
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add LLM model server metrics achandrasekar/semantic-conventions
4 participants
@AlexanderWert @achandrasekar @lmolkova @gyliu513