Code

Reuse KV cache of prefixes#484

Code

Draft

tohtana wants to merge 11 commits intomaindeepspeedai/DeepSpeed-MII:mainfrom tohtana/cache_prefixdeepspeedai/DeepSpeed-MII:tohtana/cache_prefixCopy head branch name to clipboard

+37-8

Commits

Commits on Apr 23, 2024

• reuse kv cache prefix

  tohtana

  committed
  8c90e4d

  Copy full SHA for 8c90e4d

Commits on Apr 26, 2024

• fix argument

  tohtana

  committed
  18da144

  Copy full SHA for 18da144

Commits on Apr 28, 2024

• fix alloc arg

  tohtana

  committed
  08f579d

  Copy full SHA for 08f579d

Commits on May 8, 2024

• add option to prefix cache

  tohtana

  committed
  cfb9b3a

  Copy full SHA for cfb9b3a

Commits on May 17, 2024

• flush the last request

  tohtana

  committed
  7444160

  Copy full SHA for 7444160

Commits on May 25, 2024

• add debug func

  tohtana

  committed
  42a0386

  Copy full SHA for 42a0386

Commits on May 27, 2024

• remove debug function

  tohtana

  committed
  b94bc7a

  Copy full SHA for b94bc7a

• Merge branch 'main' into tohtana/cache_prefix

  tohtana

  authored
  05a9bb8

  Copy full SHA for 05a9bb8

Commits on May 30, 2024

• save prefix cache at every iteration

  tohtana

  committed
  414f696

  Copy full SHA for 414f696

Commits on May 31, 2024

• fix token count for prefix cache

  tohtana

  committed
  263bf21

  Copy full SHA for 263bf21

Commits on Jun 3, 2024

• fix prefix cache target

  tohtana

  committed
  584407c

  Copy full SHA for 584407c