Wrong LoRA tuning example

[sofyc]

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

https://github.com/microsoft/Phi-3CookBook/blob/main/code/04.Finetuning/Phi-3-finetune-lora-python.ipynb
https://github.com/microsoft/Phi-3CookBook/blob/main/code/04.Finetuning/Phi-3-finetune-qlora-python.ipynb

The example here in these example scripts appears to be misleading, as it in peft_config, they show

target_modules = ['k_proj', 'q_proj', 'v_proj', 'o_proj', 'gate_proj', 'down_proj', 'up_proj'],

which is incorrect based on the model's architecture.

It should be
target_modules = ['qkv_proj', 'o_proj', 'gate_up_proj', 'down_proj'],

Tasks

Preview Give feedback

No tasks being tracked yet.