Huggingface quantisation

[Archit-Kohli]

• Documentation Update

• Why was this update required?: Difficult to load model with such high parameters so needs to be loaded via bitsnbytes

• Other information: Added link to hugging face's official bits-n-bytes notebook for reference

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
docs-gpt	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 4, 2023 1:56pm
nextra-docsgpt	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 4, 2023 1:56pm

[dartpain]

Unfortunately users will need to change some code in application/llm/huggingface.py
What you can do is maybe create a differnt class or a q parameter that can be passed to it called q to set quantisation if explicit by users. Will only need to add few lines of code here

basically add
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16
)

to AutoModelForCausalLM.from_pretrained and a conditional import

this would be much appreciated

[dartpain]

Unfortunately users will need to change some code in application/llm/huggingface.py
What you can do is maybe create a differnt class or a q parameter that can be passed to it called q to set quantisation if explicit by users. Will only need to add few lines of code here

[Archit-Kohli]

Updated in a new pull request #425 you can close this request