Code

llama : add Mixtral support#4406

Code

Merged

ggerganov merged 47 commits intomasterggerganov/llama.cpp:masterfrom mixtralggerganov/llama.cpp:mixtralCopy head branch name to clipboardDec 13, 2023

+2,369-394

Commits

Commits on Dec 9, 2023

• convert : support Mixtral as LLAMA arch

  ggerganov

  committed
  dff8cbe

• convert : fix n_ff typo

  ggerganov

  committed
  d38e41e

• llama : model loading

  ggerganov

  committed
  a3eefe9

• ggml : sync latest ggml_mul_mat_id

  ggerganov

  committed
  861cd67

• llama : update graph to support MoE

  ggerganov

  committed
  aedfad1

• llama : fix cur -> cur_expert

  ggerganov

  committed
  af1a096

• llama : first working version

  ggerganov

  committed
  7ea3695

• llama : fix expert weighting in the FFN

  ggerganov

  committed
  8b185b7

• ggml : ggml_get_rows support 2D indexing [n_tokens, n_experts] (cpu only)

  ggerganov

  committed
  7372b62

• ggml : add n_as argument to ggml_mul_mat_id

  slaren

  committed
  ee8fb39

• ggml : fix ggml_get_rows to take into account ne02 / ne11

  ggerganov

  committed
  9064b1c

• metal : add more general support for ggml_get_rows + tests

  ggerganov

  committed
  2cbcba8

• llama : add basic support for offloading moe with CUDA

  slaren

  committed
  06dfde3

• metal : add/mul/div use general kernel when src1 not cont

  ggerganov

  committed
  7e2006b

• metal : reduce the kernel launches for ggml_mul_mat_id

  ggerganov

  committed
  8c5b66e

• ggml : get_rows : support non-contiguos tensors with gaps, generalize up to 3D

  slaren

  committed
  ac3f7d8

• ggml : update get_rows f16 and q

  slaren

  committed
  2e4db48

• cuda : support non-contiguous src1 in get_rows

  slaren

  committed
  62b95f9

• llama : offload missing ffn_moe_silu

  slaren

  committed
  0710b0f

Commits on Dec 10, 2023

• metal : fix ggml_get_rows to work with non-cont src1

  ggerganov

  committed
  016f9bb

• metal : add indirect mat-vec kernels for all quantization types

  ggerganov

  committed
  6cfb31f

• llama : do not quantize expert gating tensors

  ggerganov

  committed
  d1259b7

• llama : add n_expert and n_expert_used to hparams + change quants

  ggerganov

  committed
  e640cbe

• test-backend-ops : add moe test

  slaren

  committed
  cefebb3

• cuda : fix get_rows when ncols is odd

  slaren

  committed
  8614aa7

• convert : determine n_ctx correctly

  ggerganov

  committed
  65923a8

• metal : fix ggml_mul_mat_id for F32

  ggerganov

  committed
  b0b83dd

• test-backend-ops : make experts more evenly probable (test_moe)

  ggerganov

  committed
  54ba263

• test-backend-ops : cleanup, add moe test for batches

  slaren

  committed
  54d254b

• test-backend-ops : add cpy from f32 -> all types test

  slaren

  committed
  f1380d7

Commits on Dec 11, 2023

• test-backend-ops : fix dequantize block offset

  slaren

  committed
  b002981

• llama : fix hard-coded number of experts

  ggerganov

  committed
  8cbaed1

• test-backend-ops : simplify and disable slow tests to avoid CI timeout

  slaren

  committed
  ffda94c

• test-backend-ops : disable MOE test with thread sanitizer

  slaren

  committed
  33e50f1

• cuda : fix mul_mat_id with multi gpu

  slaren

  committed
  296c945

• convert : use 1e6 rope_freq_base for mixtral

  slaren

  committed
  7dc75e3

• convert : fix style

  slaren

  committed
  f1cbfab

Commits on Dec 12, 2023

• convert : support safetensors format

  ggerganov

  committed
  6a419f4

• gguf-py : bump version

  slaren

  committed
  a742d9f

• metal : add cpy f16 -> f32 kernel

  ggerganov

  committed
  08eb991

• metal : fix binary ops for ne10 % 4 != 0

  ggerganov

  committed
  a51bc0c

• test-backend-ops : add one more sum_rows test

  ggerganov

  committed
  ea4402b

• ggml : do not use BLAS with ggml_mul_mat_id

  ggerganov

  committed
  90c12e6

• convert-hf : support for mixtral-instruct (#4428)

  Show description for 82e4f64
  Mrkvak

  authored
  82e4f64

Commits on Dec 13, 2023

• metal : fix soft_max kernels

  Show description for ab558ac
  ggerganov

  committed
  ab558ac

• metal : limit kernels to not use more than the allowed threads

  ggerganov

  committed
  109e7aa

• metal : switch to execution barriers + fix one of the barriers

  ggerganov

  committed
  e1241d9