[WIP] rafactor: make gqa_group_size a function argument instead of template parameter
#262
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yzh119
force-pushed
the
group-size-parameter
branch
2 times, most recently
from
34b0475 to
c4cff52
Compare
yzh119
added a commit
that referenced
this pull request
Jun 2, 2024
#262 will degrade performance because divide by group_size will be slow. This PR adds fastdiv support so that we can use faster shift and ifma operations to compute division. We copied the code from figure 10-2 in [Hacker's Delight](https://www.oreilly.com/library/view/hackers-delight-second/9780133084993/) version 2. The data structure was inspired by https://github.com/milakov/int_fastdiv/ project, we kept its license.
yzh119
force-pushed
the
group-size-parameter
branch
from
31b5e4d to
2d1b7d0
Compare
yzh119
force-pushed
the
group-size-parameter
branch
3 times, most recently
from
5e33542 to
4ee6a5b
Compare
This reverts commit 3b3ce05. wip rebase wip wip upd upd remove macros that are no longer used upd upd upd bugfix 太阳,请赐予我完成它的力量吧 fix fix remove redundant code remove conflict
yzh119
force-pushed
the
group-size-parameter
branch
from
4ee6a5b to
6baa974
Compare
|
Follow up in #301 |
Merged
yzh119
added a commit
that referenced
this pull request
Jul 10, 2024
Alibi experienced a performance degradation after #262 because of increased number of integer division. This PR fixes the issue.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Our previous implementation treats
gqa_group_sizeas a template parameter.We also have an implicit assumption that 8 should be divisible by
gqa_group_size, this is not documented and cause lots of confusion for developers.#223 could be a workaround but this would quickly increase the binary size of this library if we keep adding more and more gqa group sizes.
This PR refactors the code so that
gqa_group_sizebecomes a runtime function argument, and we don't need to compile one kernel for each of them. This would be bring very slight performance degradation togqa_group_size=1but should work good in general. This PR also removes the requirement that "8 should be divisible bygqa_group_size", so that any gqa group size should be supported.This PR should resolve #142 #181 #246 #254 #258 and so on.
Still working in progress, estimated finish time:
May 27th at noon (PST time).May 30thJune 3rd.