Code

Cuda graph#67

Code

Open

shawlleyw wants to merge 17 commits intomainUSC-NSL/DisagMoE:mainfrom cuda_graphUSC-NSL/DisagMoE:cuda_graphCopy head branch name to clipboard

+666-494

Commits

Commits on Jan 15, 2025

• remove prefill query length

  shawlleyw

  committed
  2265f0e

  Copy full SHA for 2265f0e

• rename prefill pos and prefill len as init prefill len

  shawlleyw

  committed
  06455b2

  Copy full SHA for 06455b2

Commits on Jan 16, 2025

• support init prefill length

  shawlleyw

  committed
  592110c

  Copy full SHA for 592110c

• enable input size argument

  shawlleyw

  committed
  759dc01

  Copy full SHA for 759dc01

• fix cuda graph for decoding

  shawlleyw

  committed
  9423d37

  Copy full SHA for 9423d37

Commits on Jan 17, 2025

• ix TP metadata

  shawlleyw

  committed
  cd9e3f4

  Copy full SHA for cd9e3f4

• use pinned memory for H2D

  shawlleyw

  committed
  b24a669

  Copy full SHA for b24a669

Commits on Jan 31, 2025

• permute by topk_ids in GPU

  shawlleyw

  committed
  5356aa4

  Copy full SHA for 5356aa4

Commits on Feb 4, 2025

• apply layer norm

  shawlleyw

  committed
  bbde358

  Copy full SHA for bbde358

• prepare attention batch info in c++

  shawlleyw

  committed
  61e217c

  Copy full SHA for 61e217c

Commits on Feb 5, 2025

• refactor cuda graph execution

  shawlleyw

  committed
  5e5c964

  Copy full SHA for 5e5c964

• adapt engine for cuda graph executor

  shawlleyw

  committed
  d9a19fb

  Copy full SHA for d9a19fb

Commits on Feb 6, 2025

• fix attention profile execute

  shawlleyw

  committed
  95aa50d

  Copy full SHA for 95aa50d

• fix cuda graph, still illegal memory access

  shawlleyw

  committed
  649899b

  Copy full SHA for 649899b

Commits on Feb 7, 2025

• test cuda graph before serving

  shawlleyw

  committed
  7513f94

  Copy full SHA for 7513f94

• fix cuda graph static batch infos

  shawlleyw

  committed
  3857bd7

  Copy full SHA for 3857bd7

Commits on Feb 10, 2025

• random logits & refine benchmark

  shawlleyw

  committed
  b4713a7

  Copy full SHA for b4713a7