Code

llama : support quantum K cache#4312

Code

Merged

ggerganov merged 14 commits intogg/per-layer-kvggerganov/llama.cpp:gg/per-layer-kvfrom gg/quantum-k-cacheggerganov/llama.cpp:gg/quantum-k-cacheCopy head branch name to clipboardDec 6, 2023

+552-74

Commits

Commits on Dec 3, 2023

• llama : support quantum K cache (wip)

  ggerganov

  committed
  d04ee92

Commits on Dec 4, 2023

• metal : add F32 -> Q8_0 copy kernel

  ggerganov

  committed
  bcfebf2

• cuda : add F32 -> Q8_0 copy kernel

  Show description for a1bf6c0
  ggerganov

  committed
  a1bf6c0

• cuda : use mmv kernel for quantum cache ops

  ggerganov

  committed
  b881f63

Commits on Dec 5, 2023

• llama : pass KV cache type through API

  ggerganov

  committed
  3ce30e0

• llama : fix build

  Show description for 7864a2c
  ggerganov

  committed
  7864a2c

• metal : add F32 -> Q4_0 copy kernel

  ggerganov

  committed
  9d69ecc

• metal : add F32 -> Q4_1 copy kernel

  ggerganov

  committed
  6b58ae9

• cuda : wip

  ggerganov

  committed
  e8457c9

• cuda : add F32 -> Q4_0 and F32 -> Q4_1 copy kernels

  ggerganov

  committed
  b2acede

• llama-bench : support type_k/type_v

  slaren

  committed
  903167a

• metal : use mm kernel only for quantum KV cache

  ggerganov

  committed
  dd86df8

• cuda : add comment

  ggerganov

  committed
  4adb1d6

• llama : remove memory_f16 and kv_f16 flags

  ggerganov

  committed
  af99c6f