Code

llama: implement YaRN RoPE scaling#2268

Code

Merged

cebtenzzre merged 36 commits intoggerganov:masterggerganov/llama.cpp:masterfrom cebtenzzre:ntkv2cebtenzzre/llama.cpp:ntkv2Copy head branch name to clipboardNov 1, 2023

+764-258

Commits

Commits on Aug 7, 2023

• llama: implement NTK-By-Parts (NTKv2) RoPE scaling

  cebtenzzre

  committed
  8dec38c

  Copy full SHA for 8dec38c

• CUDA implementation

  cebtenzzre

  committed
  6aeb46b

  Copy full SHA for 6aeb46b

• Metal implementation

  cebtenzzre

  committed
  9348aa4

  Copy full SHA for 9348aa4

Commits on Sep 5, 2023

• implement new YaRN algorithm

  cebtenzzre

  committed
  a30ae20

  Copy full SHA for a30ae20

Commits on Sep 21, 2023

• Merge branch 'master' of https://github.com/ggerganov/llama.cpp into ntkv2-fix

  cebtenzzre

  committed
  b5ced4f

  Copy full SHA for b5ced4f

• ggml : increase GGML_MAX_OP_PARAMS

  cebtenzzre

  committed
  826269a

  Copy full SHA for 826269a

• YaRN : avoid NaN if unused betas are zero

  cebtenzzre

  committed
  cf731d5

  Copy full SHA for cf731d5

• YaRN : fix missing parameter in CUDA impl

  cebtenzzre

  committed
  dcb058c

  Copy full SHA for dcb058c

• convert : reduce unnecessary variables in Params

  cebtenzzre

  committed
  281b26e

  Copy full SHA for 281b26e

• Merge branch 'master' of https://github.com/ggerganov/llama.cpp into ntkv2

  cebtenzzre

  committed
  a06c729

  Copy full SHA for a06c729

• llama : simplify use of context params

  cebtenzzre

  committed
  dc26a0d

  Copy full SHA for dc26a0d

• llama : store YaRN parameters in GGUF

  cebtenzzre

  committed
  904d4ed

  Copy full SHA for 904d4ed

• fix convert scripts

  cebtenzzre

  committed
  56abb9a

  Copy full SHA for 56abb9a

• llama : fix C compatibility

  cebtenzzre

  committed
  43eaf06

  Copy full SHA for 43eaf06

• don't hardcode max_pos_emb

  cebtenzzre

  committed
  fe788c4

  Copy full SHA for fe788c4

• address review comments

  cebtenzzre

  committed
  e0b120c

  Copy full SHA for e0b120c

• restore backwards compatiblity with *.rope.scale_linear

  cebtenzzre

  committed
  19bb74e

  Copy full SHA for 19bb74e

• better option descriptions in help

  cebtenzzre

  committed
  4d5fe73

  Copy full SHA for 4d5fe73

Commits on Oct 7, 2023

• gguf : store scaling type as a string instead of an int

  cebtenzzre

  committed
  7466415

  Copy full SHA for 7466415

• improve printing of YaRN parameters

  cebtenzzre

  committed
  4f4e948

  Copy full SHA for 4f4e948

• allow forcing ext_factor to zero if scaling type is YaRN

  cebtenzzre

  committed
  5d7a3a5

  Copy full SHA for 5d7a3a5

• Merge branch 'master' of https://github.com/ggerganov/llama.cpp into ntkv2

  cebtenzzre

  committed
  9bd050f

  Copy full SHA for 9bd050f

Commits on Oct 8, 2023

• fix rope_cuda parameter order

  cebtenzzre

  committed
  babf0e0

  Copy full SHA for babf0e0

• default n_yarn_orig_ctx to n_ctx_train

  cebtenzzre

  committed
  0050e1e

  Copy full SHA for 0050e1e

• fix uninitialized cparams

  cebtenzzre

  committed
  09c3102

  Copy full SHA for 09c3102

• make printed param formatting more consistent

  cebtenzzre

  committed
  57c3442

  Copy full SHA for 57c3442

Commits on Oct 11, 2023

• fix missing import

  cebtenzzre

  committed
  a20b3e6

  Copy full SHA for a20b3e6

Commits on Oct 13, 2023

• Merge branch 'master' of https://github.com/ggerganov/llama.cpp into ntkv2

  cebtenzzre

  committed
  9ef91b1

  Copy full SHA for 9ef91b1

Commits on Oct 20, 2023

• Fix YaRN inverted scaling and add "rope.scaling.type" to GGUF (#1)

  jquesnelle

  authored
  9ae10b3

  Copy full SHA for 9ae10b3

• fix YaRN ramp, make mscale conditional, add --yarn-orig-ctx (#2)

  jquesnelle

  authored
  14cf93b

  Copy full SHA for 14cf93b

Commits on Oct 22, 2023

• Merge branch 'master' of https://github.com/ggerganov/llama.cpp into ntkv2

  cebtenzzre

  committed
  237f1e7

  Copy full SHA for 237f1e7

Commits on Oct 23, 2023

• Merge branch 'master' of https://github.com/ggerganov/llama.cpp into ntkv2

  cebtenzzre

  committed
  bc8395d

  Copy full SHA for bc8395d

Commits on Oct 24, 2023

• Merge branch 'master' of https://github.com/ggerganov/llama.cpp into ntkv2

  cebtenzzre

  committed
  4d5ed83

  Copy full SHA for 4d5ed83

Commits on Oct 30, 2023

• fix loading rope.scaling.original_context_length from GGUF (#3)

  Show description for 9fc8238
  

  jquesnelle

  and

  cebtenzzre

  authored
  9fc8238

  Copy full SHA for 9fc8238

Commits on Nov 1, 2023

• implement YaRN for GPT-NeoX RoPE

  cebtenzzre

  committed
  15f26ef

  Copy full SHA for 15f26ef

• Merge branch 'master' of https://github.com/ggerganov/llama.cpp into ntkv2

  cebtenzzre

  committed
  081f738

  Copy full SHA for 081f738