How to fix the seed during Phi3.5V training.

[qwedaq]

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Thank you for your code. I would like to know how to fix the seed during training of Phi3.5V model? I have used set_seed from transformers and also fixed a seed value in training args of Trainer but none of them actually gave me the same values for two runs.

[leestott]

Ensuring reproducibility in training can be tricky, especially with complex models like Phi-3.5V. Here are some additional steps you can take to fix the seed and achieve consistent results:

1. Set Seed for All Libraries: Make sure to set the seed for all relevant libraries, including NumPy, PyTorch, and the random module.

   import random
   import numpy as np
   import torch
   from transformers import set_seed
   
   seed = 42
   random.seed(seed)
   np.random.seed(seed)
   torch.manual_seed(seed)
   torch.cuda.manual_seed_all(seed)
   set_seed(seed)

2. Ensure Deterministic Operations: Enable deterministic algorithms in PyTorch to reduce sources of randomness.

   torch.backends.cudnn.deterministic = True
   torch.backends.cudnn.benchmark = False

3. Set Seed in Training Arguments: As you mentioned, set the seed in the TrainingArguments.

   from transformers import TrainingArguments
   
   training_args = TrainingArguments(
       output_dir='./results',
       num_train_epochs=3,
       per_device_train_batch_size=4,
       per_device_eval_batch_size=4,
       warmup_steps=500,
       weight_decay=0.01,
       logging_dir='./logs',
       logging_steps=10,
       seed=seed
   )

4. Environment Variables: Sometimes, setting environment variables can help ensure reproducibility.

   import os
   
   os.environ['PYTHONHASHSEED'] = str(seed)

5. Check for Non-Deterministic Operations: Some operations in deep learning frameworks can be non-deterministic. Ensure that your code does not include such operations or that they are controlled.