Code

[CORE] Quantized lm-head Framework#4442

Code

Merged

robertgshaw2-redhat merged 177 commits intovllm-project:mainvllm-project/vllm:mainfrom Qubitium:autogptq-lm-headQubitium/vllm:autogptq-lm-headCopy head branch name to clipboardJul 2, 2024

+268-121

Commits

Commits on Apr 26, 2024

• skip unknown param g_idx for gptq

  Qubitium

  committed
  109291f

  Copy full SHA for 109291f

• skip unknown param g_idx for gptq

  Qubitium

  committed
  319eeed

  Copy full SHA for 319eeed

• skip unknown param g_idx for gptq

  Qubitium

  committed
  d17428c

  Copy full SHA for d17428c

• debug

  Qubitium

  committed
  43dc07b

  Copy full SHA for 43dc07b

• update

  Qubitium

  committed
  40dfa40

  Copy full SHA for 40dfa40

• update

  Qubitium

  committed
  b422c4c

  Copy full SHA for b422c4c

• update

  Qubitium

  committed
  cf2ef3f

  Copy full SHA for cf2ef3f

Commits on Apr 27, 2024

• test pass linear_method

  Qubitium

  committed
  75d1aad

  Copy full SHA for 75d1aad

• test pass linear_method

  Qubitium

  committed
  69963dd

  Copy full SHA for 69963dd

Commits on Apr 28, 2024

• pass linear_method

  Qubitium

  committed
  3f348ca

  Copy full SHA for 3f348ca

• req update

  Qubitium

  committed
  6392659

  Copy full SHA for 6392659

• fix issue: can't load quant lm_head weights

  

  ZX

  committed
  91b655c

  Copy full SHA for 91b655c

• call lm_head.linear_method.apply_weights()

  

  ZX

  committed
  eb4252e

  Copy full SHA for eb4252e

• Merge branch 'main' into autogptq-lm-head

  

  ZX

  committed
  4b755a1

  Copy full SHA for 4b755a1

• use quant_config.get_quant_method()

  

  ZX

  committed
  ce7f348

  Copy full SHA for ce7f348

• lm_head is VocabParallelEmbedding not LinearBase

  Qubitium

  committed
  73c956d

  Copy full SHA for 73c956d

• fix error: Overwriting existing tensor attribute: weight_loader

  

  ZX

  committed
  cc1ed62

  Copy full SHA for cc1ed62

• lm_head is VocabParallelEmbedding not LinearBase

  Qubitium

  committed
  270169e

  Copy full SHA for 270169e

• merge

  Qubitium

  committed
  5e6c796

  Copy full SHA for 5e6c796

• merge

  Show description for d667b23
  Qubitium

  committed
  d667b23

  Copy full SHA for d667b23

• set separate_bias_add=True for unquant linear_method

  Qubitium

  committed
  806b8c2

  Copy full SHA for 806b8c2

• cleanup

  Qubitium

  committed
  7bcdf89

  Copy full SHA for 7bcdf89

• try to mimic original logic

  Qubitium

  committed
  b02e1ce

  Copy full SHA for b02e1ce

• use linear_method only

  Qubitium

  committed
  2626a45

  Copy full SHA for 2626a45

• keep logic as close to main as possible

  Qubitium

  committed
  b18390f

  Copy full SHA for b18390f

• wrong name

  Qubitium

  committed
  786a49f

  Copy full SHA for 786a49f

• move linear_method decoding to ParallelLmHead class

  Qubitium

  committed
  1c2c1a5

  Copy full SHA for 1c2c1a5

• fix error: UnquantizedLinearMethod.create_weights() missing 1 required positional argument: 'params_dtype'

  

  ZX

  committed
  561b26c

  Copy full SHA for 561b26c

• Fix get embedding error

  

  ZX

  committed
  2ec6b22

  Copy full SHA for 2ec6b22

Commits on Apr 29, 2024

• revert req.txt to main

  Qubitium

  committed
  22297c1

  Copy full SHA for 22297c1

• rename

  Qubitium

  committed
  a386d0c

  Copy full SHA for a386d0c

• add QUANTIZED bool param to base LinearMethod so we can avoid all the isinstance calls

  Qubitium

  committed
  7ac901d

  Copy full SHA for 7ac901d

• wrong state

  Qubitium

  committed
  ec88f23

  Copy full SHA for ec88f23

• pep

  Qubitium

  committed
  a299189

  Copy full SHA for a299189

• override QUANTIZED property to True for all quant based linear methods

  Qubitium

  committed
  200fc77

  Copy full SHA for 200fc77

• comment TODO

  Qubitium

  committed
  7ccffee

  Copy full SHA for 7ccffee

• pass quant_config to base class and init linear_method there

  Qubitium

  committed
  7247aa7

  Copy full SHA for 7247aa7

• ruff

  Qubitium

  committed
  b343695

  Copy full SHA for b343695

• ruff

  Qubitium

  committed
  1b07243

  Copy full SHA for 1b07243

• de-ruff

  Qubitium

  committed
  49b0fa0

  Copy full SHA for 49b0fa0

• revert newline

  Qubitium

  committed
  dd5835d

  Copy full SHA for dd5835d

• fix bad class name QuantizeMethodBase to QuantizableMethodBase

  Qubitium

  committed
  ea6819b

  Copy full SHA for ea6819b

• Merge branch 'main' into autogptq-lm-head

  Qubitium

  authored
  a51b26e

  Copy full SHA for a51b26e

• The output_size parameter of create_weights() should be total embeddings

  

  ZX

  committed
  7866461

  Copy full SHA for 7866461

• missed rename

  Qubitium

  committed
  018b5a5

  Copy full SHA for 018b5a5

• isort

  Qubitium

  committed
  740a70b

  Copy full SHA for 740a70b

• clean

  Qubitium

  committed
  b3b6187

  Copy full SHA for b3b6187

• modify weight_loader()

  

  ZX

  committed
  ea2f9eb

  Copy full SHA for ea2f9eb

• format

  Qubitium

  committed
  aab9566

  Copy full SHA for aab9566

• short method name and apply to all models

  Qubitium

  committed
  52b6aba

  Copy full SHA for 52b6aba

• move method into correct util

  Qubitium

  committed
  8a54dca

  Copy full SHA for 8a54dca

• add test_quant_lm_head_layer()

  

  ZX

  committed
  8fb68a2

  Copy full SHA for 8fb68a2

• pass lm_head and not lm_head.weights

  Qubitium

  committed
  193d093

  Copy full SHA for 193d093

• Merge remote-tracking branch 'origin/autogptq-lm-head' into autogptq-lm-head

  Qubitium

  committed
  894ee7b

  Copy full SHA for 894ee7b

• add test_quant_lm_head_false()

  

  ZX

  committed
  9773b41

  Copy full SHA for 9773b41

• do not change opt/lm_head type

  Qubitium

  committed
  e1a8ec4

  Copy full SHA for e1a8ec4

• Merge remote-tracking branch 'origin/autogptq-lm-head' into autogptq-lm-head

  Qubitium

  committed
  950d9da

  Copy full SHA for 950d9da

• pass test_logits_processor

  Qubitium

  committed
  e83a0c0

  Copy full SHA for e83a0c0

• enforce_eager for more stable ci test

  Qubitium

  committed
  f3cdff9

  Copy full SHA for f3cdff9

• revert model lm_head type to original

  Qubitium

  committed
  0c18da0

  Copy full SHA for 0c18da0

• standardize lm_head for ParallelLMHead

  Qubitium

  committed
  1e03ad8

  Copy full SHA for 1e03ad8

• ruff/import

  Qubitium

  committed
  a120090

  Copy full SHA for a120090

• format

  Qubitium

  committed
  3229b6c

  Copy full SHA for 3229b6c

• Revert "standardize lm_head for ParallelLMHead"

  Show description for ccd0d54
  Qubitium

  committed
  ccd0d54

  Copy full SHA for ccd0d54

• fix oom in test

  Qubitium

  committed
  0c9b5d4

  Copy full SHA for 0c9b5d4

• fix starcoder2

  Qubitium

  committed
  6a857b4

  Copy full SHA for 6a857b4

• fix lora

  Qubitium

  committed
  90c70d2

  Copy full SHA for 90c70d2

• fix lora test

  Qubitium

  committed
  b3f3670

  Copy full SHA for b3f3670

• pass quant_config to all ParallelLMHead

  Qubitium

  committed
  65e9ceb

  Copy full SHA for 65e9ceb

Commits on Apr 30, 2024

• comment

  Qubitium

  committed
  9716863

  Copy full SHA for 9716863

• expand testing for refactor

  robertgshaw2-redhat

  committed
  7b72ef0

  Copy full SHA for 7b72ef0

• Merge remote-tracking branch 'upstream/main' into expand-testing

  robertgshaw2-redhat

  committed
  83a0087

  Copy full SHA for 83a0087

• expanded testing

  robertgshaw2-redhat

  committed
  18f8f35

  Copy full SHA for 18f8f35

• raise error if lm_head is quantized but layer is not ParallelLMHead

  Qubitium

  committed
  fd1cf0e

  Copy full SHA for fd1cf0e

• merge lm_false and _true into one test file

  

  ZX

  committed
  da0af51

  Copy full SHA for da0af51

• formatting

  robertgshaw2-redhat

  committed
  589b50d

  Copy full SHA for 589b50d

• format

  Qubitium

  committed
  18e6996

  Copy full SHA for 18e6996

• Merge pull request #2 from neuralmagic/expand-testing

  Show description for 97c2870
  Qubitium

  authored
  97c2870

  Copy full SHA for 97c2870

• Revert "raise error if lm_head is quantized but layer is not ParallelLMHead"

  Show description for e3414df
  Qubitium

  committed
  e3414df

  Copy full SHA for e3414df

• fix test

  Qubitium

  committed
  81cc15c

  Copy full SHA for 81cc15c

• Skip loading of weights of lm_head layer.

  Show description for 232c09d
  

  ZX

  committed
  232c09d

  Copy full SHA for 232c09d

• test for opt-125M

  

  ZX

  committed
  cd14c7a

  Copy full SHA for cd14c7a

• rename VocabParallelEmbedding to ParallelVocabEmbedding for consistency

  Qubitium

  committed
  1f0ad4c

  Copy full SHA for 1f0ad4c

• Merge remote-tracking branch 'origin/autogptq-lm-head' into autogptq-lm-head

  Qubitium

  committed
  83f3823

  Copy full SHA for 83f3823

• fix opt not loading quantized lm_head

  Qubitium

  committed
  d94e6f8

  Copy full SHA for d94e6f8

• Merge branch 'main' into autogptq-lm-head

  robertgshaw2-redhat

  authored
  d5a3667

  Copy full SHA for d5a3667

• updated test, opt model is failing

  robertgshaw2-redhat

  committed
  8fde7f5

  Copy full SHA for 8fde7f5

Commits on May 1, 2024

• fix opt and disable marlin when lm_head quant is enabled. TODO find out why marlin load is broken

  Qubitium

  committed
  2f63a72

  Copy full SHA for 2f63a72

• Merge remote-tracking branch 'origin/autogptq-lm-head' into autogptq-lm-head

  Show description for 4bdf676
  Qubitium

  committed
  4bdf676

  Copy full SHA for 4bdf676

• remove debug prints

  Qubitium

  committed
  fd0a758

  Copy full SHA for fd0a758

• isort

  Qubitium

  committed
  7ef2060

  Copy full SHA for 7ef2060

• add non-quantized opt-125m to test to ensure that quant code did not break non-quant

  Qubitium

  committed
  5b8bba9

  Copy full SHA for 5b8bba9

• 1. move lm_head_quantized proper into base QuantizationConfig class.

  Show description for 3a461ec
  Qubitium

  committed
  3a461ec

  Copy full SHA for 3a461ec

• format

  Qubitium

  committed
  5d30ab8

  Copy full SHA for 5d30ab8

Commits on May 3, 2024

• fix marlin loading of lm_head. remove TODO

  Qubitium

  committed
  09cca74

  Copy full SHA for 09cca74

Commits on May 6, 2024

• add comments for weight_loader

  

  ZX

  committed
  2097e0d

  Copy full SHA for 2097e0d

• Replace BaseConfig.is_lm_head_quantized property with BaseConfig.is_lm_head_quantized() interface method

  Qubitium

  committed
  9981620

  Copy full SHA for 9981620

• Merge remote-tracking branch 'origin/autogptq-lm-head' into autogptq-lm-head

  Qubitium

  committed
  a63c8a3

  Copy full SHA for a63c8a3

• implement BaseConfig.is_lm_head_quantized() for all non-gptq quant configs

  Qubitium

  committed
  f8f705e

  Copy full SHA for f8f705e

• fix lora logits bias computed twice

  Qubitium

  committed
  1e8bff6

  Copy full SHA for 1e8bff6

• format

  Qubitium

  committed
  d525139

  Copy full SHA for d525139

Commits on Jun 29, 2024

• merged - removed quantized lm head for tied embeddings

  

  [email protected]

  committed
  282a703

  Copy full SHA for 282a703

• format

  

  [email protected]

  committed
  770b091

  Copy full SHA for 770b091

• format passing

  

  [email protected]

  committed
  51b0064

  Copy full SHA for 51b0064

• fix all parallel-vocab-embedding --> vocab-parallel-embedding

  

  [email protected]

  committed
  6deae36

  Copy full SHA for 6deae36

• quantizable-method-base --> quantize-method-base

  

  [email protected]

  committed
  5828457

  Copy full SHA for 5828457

• spurious change

  

  [email protected]

  committed
  c0551d5

  Copy full SHA for c0551d5

• cleanup AQLM

  

  [email protected]

  committed
  732971e

  Copy full SHA for 732971e

• missed nit

  

  [email protected]

  committed
  584d644

  Copy full SHA for 584d644

• fix up awq

  

  [email protected]

  committed
  eda6d40

  Copy full SHA for eda6d40

• fix up fp8

  

  [email protected]

  committed
  37ede38

  Copy full SHA for 37ede38

• nit

  

  [email protected]

  committed
  df5d5b5

  Copy full SHA for df5d5b5

• cleanup gptq marlin

  

  [email protected]

  committed
  c0cf2fa

  Copy full SHA for c0cf2fa

• cleanup more

  

  [email protected]

  committed
  43c67f1

  Copy full SHA for 43c67f1

• nit

  

  [email protected]

  committed
  746fd43

  Copy full SHA for 746fd43

• remove QUANTIZED

  

  [email protected]

  committed
  800d3f7

  Copy full SHA for 800d3f7

• if -- elif

  

  [email protected]

  committed
  5b0a580

  Copy full SHA for 5b0a580

• removed

  

  [email protected]

  committed
  37795a6

  Copy full SHA for 37795a6

• add kwarg

  

  [email protected]

  committed
  ac5494e

  Copy full SHA for ac5494e

• cleanup

  

  [email protected]

  committed
  e8e8612

  Copy full SHA for e8e8612

• more cleanup

  

  [email protected]

  committed
  5bbdb95

  Copy full SHA for 5bbdb95

• nit

  

  [email protected]

  committed
  66b07b1

  Copy full SHA for 66b07b1

• use linear_method.create_weights

  

  [email protected]

  committed
  d755b4c

  Copy full SHA for d755b4c

• fix spurious changes

  

  [email protected]

  committed
  4fc9de5

  Copy full SHA for 4fc9de5

• fix typo

  

  [email protected]

  committed
  9a699b5

  Copy full SHA for 9a699b5

• remove another spurious change

  

  [email protected]

  committed
  58bbf5d

  Copy full SHA for 58bbf5d

• format

  

  [email protected]

  committed
  656ae2d

  Copy full SHA for 656ae2d

Commits on Jun 30, 2024

• format fix

  

  [email protected]

  committed
  90ecd98

  Copy full SHA for 90ecd98

• passing linting

  

  [email protected]

  committed
  1916467

  Copy full SHA for 1916467

• fix more changes

  

  [email protected]

  committed
  8981bcd

  Copy full SHA for 8981bcd

• spurious newline

  

  [email protected]

  committed
  9e38196

  Copy full SHA for 9e38196

• spurious change in qwen

  

  [email protected]

  committed
  b3ad6d5

  Copy full SHA for b3ad6d5

• first end to end run of llama working!

  

  [email protected]

  committed
  db182dd

  Copy full SHA for db182dd

• format

  

  [email protected]

  committed
  20a7fbf

  Copy full SHA for 20a7fbf

• cleanup

  

  [email protected]

  committed
  90ef065

  Copy full SHA for 90ef065

• Merge branch 'main' into autogptq-lm-head

  

  [email protected]

  committed
  1990fff

  Copy full SHA for 1990fff

• format

  

  [email protected]

  committed
  8f3dae8

  Copy full SHA for 8f3dae8

• fix opt

  

  [email protected]

  committed
  a8c5aeb

  Copy full SHA for a8c5aeb

• fix gpt2

  

  [email protected]

  committed
  d29ba07

  Copy full SHA for d29ba07

• fix arctic

  

  [email protected]

  committed
  47cd399

  Copy full SHA for 47cd399

• fix gptbigcode

  

  [email protected]

  committed
  92f8eb4

  Copy full SHA for 92f8eb4

• actually fix gptbigcode

  

  [email protected]

  committed
  03db82c

  Copy full SHA for 03db82c

• commandr

  

  [email protected]

  committed
  c141c48

  Copy full SHA for c141c48

• fix deepseekv2

  

  [email protected]

  committed
  69f606a

  Copy full SHA for 69f606a

• fix falcon

  

  [email protected]

  committed
  85e4414

  Copy full SHA for 85e4414

• fix gemma2

  

  [email protected]

  committed
  e04823d

  Copy full SHA for e04823d

• fix jais

  

  [email protected]

  committed
  1ada5c0

  Copy full SHA for 1ada5c0

• fix mpt

  

  [email protected]

  committed
  8666f8b

  Copy full SHA for 8666f8b

• fix phi3 small

  

  [email protected]

  committed
  47b6b64

  Copy full SHA for 47b6b64

• fix vision models

  

  [email protected]

  committed
  1e85d83

  Copy full SHA for 1e85d83

• all models should run

  

  [email protected]

  committed
  a7ebdb2

  Copy full SHA for a7ebdb2

• formatted

  

  [email protected]

  committed
  8a41ff2

  Copy full SHA for 8a41ff2

• cleanup lora test changes

  

  [email protected]

  committed
  9f8cf92

  Copy full SHA for 9f8cf92

• cleanup tests

  

  [email protected]

  committed
  31ec9e3

  Copy full SHA for 31ec9e3

• nit

  

  [email protected]

  committed
  280b4cc

  Copy full SHA for 280b4cc

• remove llama debug

  

  [email protected]

  committed
  f146569

  Copy full SHA for f146569

• fix spurious change in qwen2

  

  [email protected]

  committed
  4fed894

  Copy full SHA for 4fed894

• formats again

  

  [email protected]

  committed
  29a1fb1

  Copy full SHA for 29a1fb1

• re-enabled tests

  

  [email protected]

  committed
  57bc23a

  Copy full SHA for 57bc23a

• format

  

  [email protected]

  committed
  fe87469

  Copy full SHA for fe87469

• skip fp16 tests in non cuda automation

  

  [email protected]

  committed
  40fac6a

  Copy full SHA for 40fac6a

• fix mlp-speculator

  

  [email protected]

  committed
  ab2c6a9

  Copy full SHA for ab2c6a9

• fix spec decode tests

  

  [email protected]

  committed
  a0e937a

  Copy full SHA for a0e937a

• passing mlp correctness with precision of float32

  

  [email protected]

  committed
  28fcecf

  Copy full SHA for 28fcecf

• Merge branch 'upstream-main' into autogptq-lm-head

  robertgshaw2-redhat

  committed
  2ab00c0

  Copy full SHA for 2ab00c0

• Merge branch 'upstream-main' into autogptq-lm-head

  robertgshaw2-redhat

  committed
  d8306f7

  Copy full SHA for d8306f7

• format

  robertgshaw2-redhat

  committed
  c0ce169

  Copy full SHA for c0ce169

Commits on Jul 1, 2024

• Merge branch 'main' into autogptq-lm-head

  robertgshaw2-redhat

  authored
  09723d9

  Copy full SHA for 09723d9

• Address cody's comments

  robertgshaw2-redhat

  committed
  631f3a5

  Copy full SHA for 631f3a5

• Merge branch 'autogptq-lm-head' of https://github.com/Qubitium/vllm into autogptq-lm-head

  robertgshaw2-redhat

  committed
  3a83edb

  Copy full SHA for 3a83edb

• format

  robertgshaw2-redhat

  committed
  f1f9f48

  Copy full SHA for f1f9f48

• updated names

  robertgshaw2-redhat

  committed
  3eae1ad

  Copy full SHA for 3eae1ad

• format

  robertgshaw2-redhat

  committed
  ef94034

  Copy full SHA for ef94034

• Merge branch 'main' into autogptq-lm-head

  robertgshaw2-redhat

  authored
  a5a1adf

  Copy full SHA for a5a1adf

Commits on Jul 2, 2024

• remove too long tests

  robertgshaw2-redhat

  committed
  dfafb07

  Copy full SHA for dfafb07

• cleanup tests

  robertgshaw2-redhat

  committed
  76633ed

  Copy full SHA for 76633ed

• updated

  robertgshaw2-redhat

  committed
  15b3865

  Copy full SHA for 15b3865