Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix the rushed out multi-query kernel #44

Closed
zhuohan123 opened this issue Apr 22, 2023 · 2 comments
Closed

Fix the rushed out multi-query kernel #44

zhuohan123 opened this issue Apr 22, 2023 · 2 comments
Assignees
@WoosukKwon

Comments

@zhuohan123
Copy link
Member

  1. Fix the correctness issue in the current FlashAttention-copy-based kernel. Make sure we call the FlashAttention kernel correctly. Evaluate the performance of this kernel.
  2. Reduce the memory usage of the current kernel by limiting the buffer size and calling the kernel multiple times.
@WoosukKwon WoosukKwon self-assigned this May 2, 2023
@zhuohan123 zhuohan123 mentioned this issue Jun 25, 2023
Closed
76 tasks
@tmm1
Copy link
Contributor

tmm1 commented Aug 3, 2023

current FlashAttention-copy-based kernel

I believe this is referring to #4, however as of #70 flash-attn is no longer used.

@shanshanpt shanshanpt mentioned this issue Nov 17, 2023
Closed
@junior-zsy junior-zsy mentioned this issue Nov 20, 2023
Closed
@hmellor
Copy link
Collaborator

hmellor commented Mar 8, 2024

Closing based on @tmm1's comment about flash-attn no longer being used.

@hmellor hmellor closed this as completed Mar 8, 2024
@yuhuixu1993 yuhuixu1993 mentioned this issue Jun 2, 2024
Closed
tianyil1 pushed a commit to tianyil1/vllm that referenced this issue Jun 5, 2024
@madamczykhabana
Graphs v2 (vllm-project#44)
b3617ee 
* Trimmed metadata - part 1

* [WIP] HPU graphs for decode

* [WIP] Graph allocation algorithm reworked

* Cleanup

* Add graph memory estimations

* Fix multinode synchronization

* Create attn_bias inside HPU graph

* Cleanup after rebase

* Increase default VLLM_GRAPH_RESERVED_MEM to 0.3

* Remove obsolete class

* Tweak default HPU graph parameters
fxmarty pushed a commit to fxmarty/vllm-public that referenced this issue Jun 12, 2024
@charlifu @mawong-amd
Adding fp8 to gradlib (vllm-project#44)
d254de7 
* adding fp8 gemm tunner to gradlib

* formatting

* add instructions

* Linting

* adding fp8 gemm tunner to gradlib

formatting

add instructions

* Linting fp8 gradlib

* fix merging issue of ROCm_performance.md

* delete fp8_gemm_tuner.py

* Fix linting for triton: unmeld if with constexpr

* update tutorial

* Fix linting again

* fix typo

---------

Co-authored-by: Matthew Wong <[email protected]>
yukavio pushed a commit to yukavio/vllm that referenced this issue Jul 3, 2024
@mgoin
Update setup.py naming (vllm-project#44)
a477771
@ZHJ19970917 ZHJ19970917 mentioned this issue Jul 14, 2024
Closed
@alixiaodi alixiaodi mentioned this issue Aug 2, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@WoosukKwon WoosukKwon

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@tmm1 @zhuohan123 @hmellor @WoosukKwon