Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[kernel] Use sgl_kernel rope #3169

Merged
merged 10 commits into from
Jan 28, 2025
Merged

[kernel] Use sgl_kernel rope #3169

merged 10 commits into from
Jan 28, 2025

Conversation

ByronHsu
Copy link
Collaborator

@ByronHsu ByronHsu commented Jan 27, 2025

Motivation

Depends on https://github.com/sgl-project/sglang/actions/runs/12984127304

Modifications

Checklist

@ByronHsu ByronHsu marked this pull request as ready for review January 27, 2025 07:58
@zhyncs
Copy link
Member

zhyncs commented Jan 27, 2025

Please update this version limit.

"sgl-kernel>=0.0.2.post18", "torch", "vllm==0.6.4.post1",

@ByronHsu
Copy link
Collaborator Author

needs #3173

self.is_neox_style,
)
if _is_cuda_available:
apply_rope_with_cos_sin_cache_inplace(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything is going well except for an accuracy issue with test_session_control.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect the test is flaky by nature. I switched from 1B to 8B model on the main and test_session_control failed with different output: #3184.

Here is my finding so far

Kernel 1B Model 8B Model
vLLM Pass Fail
flashinfer Fail Fail

Given other tests for accuracy all pass. I think the correctness looks ok

@ByronHsu ByronHsu force-pushed the byhsu/use-fi-rope branch 3 times, most recently from 2bf7531 to 826e2e5 Compare January 28, 2025 02:24
@zhyncs zhyncs merged commit 988d0a4 into main Jan 28, 2025
4 of 16 checks passed
@zhyncs zhyncs deleted the byhsu/use-fi-rope branch January 28, 2025 06:33
@zhyncs
Copy link
Member

zhyncs commented Jan 28, 2025

Byron is cooking!! Great work.

@ByronHsu
Copy link
Collaborator Author

"Byron" XD

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants