Issue in fine-tuning Embedding models in Sentence Transformers with newer versions

[dipanjanS]

TypeError: SentenceTransformerTrainer.compute_loss() got an unexpected keyword argument 'num_items_in_batch'

This error keeps popping up especially after Hugging Face made some major changes I am assuming after 4.44 where it was still working

Versions:

sentence-transformers-3.2.1
transformers-4.46.0

Code snippet which worked fine earlier

Versions which worked:

sentence-transformers-3.0.1
transformers-4.44

from sentence_transformers import SentenceTransformer
from sentence_transformers.losses import MultipleNegativesRankingLoss
from sentence_transformers import SentenceTransformerTrainingArguments
from sentence_transformers.training_args import BatchSamplers
from sentence_transformers import SentenceTransformerTrainer

model = SentenceTransformer(
    "BAAI/bge-base-en-v1.5",
)

args = SentenceTransformerTrainingArguments(
    # Required parameter:
    output_dir="bge-base-runs",
    # Optional training parameters:
    max_steps=332,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    learning_rate=3e-6,
    warmup_ratio=0.1,
    fp16=True,  # Set to False if you get an error that your GPU can't run on FP16
    bf16=False,  # Set to True if you have a GPU that supports BF16
    batch_sampler=BatchSamplers.NO_DUPLICATES,  # MultipleNegativesRankingLoss benefits from no duplicate samples in a batch
    # Optional tracking/debugging parameters:
    eval_strategy="steps",
    eval_steps=20,
    save_strategy="steps",
    save_steps=100,
    save_total_limit=2,
    logging_steps=20,
)

loss = MultipleNegativesRankingLoss(model)

trainer = SentenceTransformerTrainer(
    model=model,
    args=args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    loss=loss,
)

trainer.train()

Error Trace:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[18], line 1
----> 1 trainer.train()

File /usr/local/lib/python3.10/dist-packages/transformers/trainer.py:2122, in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   2120         hf_hub_utils.enable_progress_bars()
   2121 else:
-> 2122     return inner_training_loop(
   2123         args=args,
   2124         resume_from_checkpoint=resume_from_checkpoint,
   2125         trial=trial,
   2126         ignore_keys_for_eval=ignore_keys_for_eval,
   2127     )

File /usr/local/lib/python3.10/dist-packages/transformers/trainer.py:2474, in Trainer._inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
   2471     self.control = self.callback_handler.on_step_begin(args, self.state, self.control)
   2473 with self.accelerator.accumulate(model):
-> 2474     tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
   2476 if (
   2477     args.logging_nan_inf_filter
   2478     and not is_torch_xla_available()
   2479     and (torch.isnan(tr_loss_step) or torch.isinf(tr_loss_step))
   2480 ):
   2481     # if loss is nan or inf simply add the average of previous logged losses
   2482     tr_loss = tr_loss + tr_loss / (1 + self.state.global_step - self._globalstep_last_logged)

File /usr/local/lib/python3.10/dist-packages/transformers/trainer.py:3572, in Trainer.training_step(self, model, inputs, num_items_in_batch)
   3569     return loss_mb.reduce_mean().detach().to(self.args.device)
   3571 with self.compute_loss_context_manager():
-> 3572     loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch)
   3574 del inputs
   3575 if (
   3576     self.args.torch_empty_cache_steps is not None
   3577     and self.state.global_step % self.args.torch_empty_cache_steps == 0
   3578 ):

TypeError: SentenceTransformerTrainer.compute_loss() got an unexpected keyword argument 'num_items_in_batch'

@tomaarsen I'm not sure if I changed anything but I've been using the above code which works great so far but tried with newer versions today and faced this.

[BelisaDi]

Hi, I've been having the same error since today. I updated both transformers and sentence-transformers, but still the same error.

Edit: Had to downgrade to transformers==4.45.2 sentence-transformers==3.1.1 to be able to finetune again.

[dipanjanS]

Yep. It's probably because transformers has changed some stuff between 4.44 and 4.46. I reinstalled the older versions and it still works.

…

On Fri, Oct 25, 2024, 1:03 AM Isabella Martínez ***@***.***> wrote: Hi, I've been having the same error since today. I updated both transformers and sentence-transformers, but still the same error. — Reply to this email directly, view it on GitHub <#3021 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2J3R54MVVB2RFDQYSWDA3Z5FDQDAVCNFSM6AAAAABQRQVZOOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZWGE4TANZRHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>