Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

It seems to be an error when getting_peft_model:ValueError: not enough values to unpack (expected 2, got 1) #14

Open
Pashisfisuta opened this issue Jun 19, 2024 · 1 comment

Comments

@Pashisfisuta
Copy link

Traceback (most recent call last):
File "/ossfs/workspace/sft/sft_all.py", line 161, in
train()
File "/ossfs/workspace/sft/sft_all.py", line 125, in train
Traceback (most recent call last):
File "/ossfs/workspace/sft/sft_all.py", line 161, in
Traceback (most recent call last):
File "/ossfs/workspace/sft/sft_all.py", line 161, in
Traceback (most recent call last):
File "/ossfs/workspace/sft/sft_all.py", line 161, in
model = get_peft_model(model, peft_config)
File "/opt/conda/lib/python3.8/site-packages/peft/mapping.py", line 149, in get_peft_model
return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](model, peft_config, adapter_name=adapter_name)
File "/opt/conda/lib/python3.8/site-packages/peft/peft_model.py", line 1395, in init
super().init(model, peft_config, adapter_name)
File "/opt/conda/lib/python3.8/site-packages/peft/peft_model.py", line 138, in init
self.base_model = cls(model, {adapter_name: peft_config}, adapter_name)
File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/model.py", line 139, in init
super().init(model, config, adapter_name)
File "/opt/conda/lib/python3.8/site-packages/peft/tuners/tuners_utils.py", line 166, in init
self.inject_adapter(self.model, adapter_name)
File "/opt/conda/lib/python3.8/site-packages/peft/tuners/tuners_utils.py", line 372, in inject_adapter
self._create_and_replace(peft_config, adapter_name, target, target_name, parent, current_key=key)
File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/model.py", line 223, in _create_and_replace
new_module = self._create_new_module(lora_config, adapter_name, target, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/model.py", line 314, in _create_new_module
new_module = dispatcher(target, adapter_name, lora_config=lora_config, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/layer.py", line 1116, in dispatch_default
new_module = Linear(target, adapter_name, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/layer.py", line 410, in init
self.update_layer(
File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/layer.py", line 117, in update_layer
self.pissa_init(adapter_name, init_lora_weights)
File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/layer.py", line 179, in pissa_init
Vr, Sr, Ur = svd_lowrank(
File "/opt/conda/lib/python3.8/site-packages/torch/_lowrank.py", line 137, in svd_lowrank
Traceback (most recent call last):
File "/ossfs/workspace/sft/sft_all.py", line 161, in
return _svd_lowrank(A, q=q, niter=niter, M=M)
File "/opt/conda/lib/python3.8/site-packages/torch/_lowrank.py", line 147, in _svd_lowrank
m, n = A.shape[-2:]
ValueError: not enough values to unpack (expected 2, got 1)

@Pashisfisuta
Copy link
Author

Pashisfisuta commented Jun 19, 2024

more information:
peft==0.11.1
load_model_code :

model = AutoModelForCausalLM.from_pretrained(
model_args.model_name_or_path,
torch_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained(
model_args.model_name_or_path,
use_fast=False,
)
tokenizer.pad_token_id = tokenizer.eos_token_id

peft_config = LoraConfig(
    r=8,
    lora_alpha=16,
    lora_dropout=0.1,
    init_lora_weights="pissa_niter_4",
    target_modules=["q_proj", "o_proj", "k_proj", "v_proj", "gate_proj", "up_proj", "down_proj"],
    bias="none",
    task_type="CAUSAL_LM",
)

model.enable_input_require_grads()
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant