Code

[Kernel] Add w8a8 CUTLASS kernels#4749

Code

Merged

robertgshaw2-redhat merged 20 commits intovllm-project:mainvllm-project/vllm:mainfrom neuralmagic:tms/w8a8_cutlass_kernelsneuralmagic/nm-vllm:tms/w8a8_cutlass_kernelsCopy head branch name to clipboardMay 16, 2024

+1,197-2

Commits

Commits on May 16, 2024

• Hopper Int and FP8 kernel for per tensor or per row/col

  tlrmchlsmth

  committed
  aa13b89

  Copy full SHA for aa13b89

• 2.x kernel for Ampere and Lovelace

  tlrmchlsmth

  committed
  d2fef83

  Copy full SHA for d2fef83

• Refactor, cleanup

  tlrmchlsmth

  committed
  24358c4

  Copy full SHA for 24358c4

• format

  tlrmchlsmth

  committed
  9e746c5

  Copy full SHA for 9e746c5

• fixup

  tlrmchlsmth

  committed
  15c46ca

  Copy full SHA for 15c46ca

• fixup

  tlrmchlsmth

  committed
  73fe18a

  Copy full SHA for 73fe18a

• Only compile the cutlass kernel for sm90a. Saves 17 MB

  tlrmchlsmth

  committed
  2231691

  Copy full SHA for 2231691

• add comment

  tlrmchlsmth

  committed
  a4c88df

  Copy full SHA for a4c88df

• Optimizations for Ampere, scalar broadcast checks

  tlrmchlsmth

  committed
  47e95ad

  Copy full SHA for 47e95ad

• review comments

  tlrmchlsmth

  committed
  62a252b

  Copy full SHA for 62a252b

• Support SM75 and fp16 output types

  tlrmchlsmth

  committed
  cdb8e06

  Copy full SHA for cdb8e06

• format.sh

  tlrmchlsmth

  committed
  5be8af7

  Copy full SHA for 5be8af7

• use cudaDeviceGetAttribute

  tlrmchlsmth

  committed
  ca43e2c

  Copy full SHA for ca43e2c

• Review comments

  tlrmchlsmth

  committed
  d09d973

  Copy full SHA for d09d973

• fixups, cruft

  tlrmchlsmth

  committed
  9a07760

  Copy full SHA for 9a07760

• review comments

  tlrmchlsmth

  committed
  4909784

  Copy full SHA for 4909784

• assert -> TORCH_CHECK

  tlrmchlsmth

  committed
  26f5890

  Copy full SHA for 26f5890

• Merge remote-tracking branch 'origin/upstream-main' into tms/w8a8_cutlass_kernels

  tlrmchlsmth

  committed
  ee1c11b

  Copy full SHA for ee1c11b

• format

  tlrmchlsmth

  committed
  39b838f

  Copy full SHA for 39b838f

• Merge branch 'upstream-main' into tms/w8a8_cutlass_kernels

  tlrmchlsmth

  committed
  7099361

  Copy full SHA for 7099361