Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Rotatory Positional Encoding #215

Merged
merged 35 commits into from
Nov 22, 2024
Merged
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
504acfc
start RoPE
abourramouss Nov 12, 2024
50aab6c
Merge branch 'hikettei:main' into main
abourramouss Nov 12, 2024
0fb82c6
Update positional-encoding.lisp
abourramouss Nov 12, 2024
dde647e
Update positional-encoding.lisp
abourramouss Nov 13, 2024
d250dc3
Update positional-encoding.lisp
abourramouss Nov 13, 2024
18106ca
x1 and x2 pairs for rope
abourramouss Nov 13, 2024
8bd08e0
remove loop, correct view usage
abourramouss Nov 14, 2024
9374cbb
single let block
abourramouss Nov 14, 2024
401bf9f
final embedding shape issues
abourramouss Nov 14, 2024
e7cd6d2
remove expansion
abourramouss Nov 14, 2024
1c4f0fb
restore expansion, it works!
abourramouss Nov 14, 2024
128708a
started test
abourramouss Nov 15, 2024
57c5fc5
feat: test
abourramouss Nov 18, 2024
ad0c3d2
test and corrections to rope
abourramouss Nov 19, 2024
db9d800
Merge branch 'hikettei:main' into main
abourramouss Nov 20, 2024
990a9ca
't = t
abourramouss Nov 20, 2024
632adb3
!rope to nn
abourramouss Nov 20, 2024
98e03d8
!rope to nn
abourramouss Nov 20, 2024
32ed373
remove proceed from rope call
abourramouss Nov 20, 2024
3d6824e
add file to test-suite
abourramouss Nov 20, 2024
556ef60
export !rope
abourramouss Nov 20, 2024
a619956
fix freqs
abourramouss Nov 20, 2024
b1eddc6
Update test-rope.lisp
abourramouss Nov 20, 2024
fb3ad75
Update positional-encoding.lisp
abourramouss Nov 20, 2024
400bb19
new RoPE implementation based on torch
abourramouss Nov 21, 2024
a88cda9
Update test-rope.lisp
abourramouss Nov 21, 2024
d708108
Update positional-encoding.lisp
abourramouss Nov 21, 2024
a15d4ea
add dim param to !rope
abourramouss Nov 22, 2024
f3154c6
Update test-rope.lisp
abourramouss Nov 22, 2024
7efffa9
use slot dim and add check
abourramouss Nov 22, 2024
75aa937
Merge branch 'main' into main
hikettei Nov 22, 2024
688ab62
review changes
abourramouss Nov 22, 2024
9233d15
Merge branch 'main' of https://github.com/abourramouss/Caten
abourramouss Nov 22, 2024
71f7dfb
add types
abourramouss Nov 22, 2024
ff2ba05
Merge branch 'main' into main
hikettei Nov 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

These merge commits were added into this branch cleanly.

There are no new changes to show.