[Launch Blocker] The default --quantize config/data/desktop.json is slow with eager, compile and aoti

[mergennachin]

One of the action items of this issue #621 was to have a default desktop.json quantization config. Since that was added, I tried running using the said config. It was slow.

1. Eager:

python3 torchchat.py generate llama3 --quantize config/data/desktop.json --prompt "Hello, my name is"

Average tokens/sec: 0.67

2. Eager+Compile:

python3 torchchat.py generate llama3 --quantize config/data/desktop.json --prompt "Hello, my name is" --compile

Average tokens/sec: 0.60

3. AOTI:

python3 torchchat.py generate llama3 --quantize config/data/desktop.json --dso-path llama3.so --prompt "Hello my name is"

Average tokens/sec: (pending - It is currently taking a long time for the first byte)

The default desktop.json setting should be performant. An action item is to either change the desktop.json to have the right config that is fast or make the execution to be performant.

Setup:

git commit: 4a83474
python version: 3.10.0
macbook pro M1

Internal Task: T187941181

[mikekgfb]

OK, so we can fix this for now, by removing quantization because that won't be accelerated until we move the pytorch pin.
Temporary fix #664

[mikekgfb]

Please confirm fix and close if resolved. @mergennachin

[mergennachin]

Please confirm fix and close if resolved.

Let's keep it open until we move the pytorch pin to include MPS kernels. I will run the commands again on the new pin. @mikekgfb