[WIP] Unify Policy Trainers#1586

Closed

lapp0 wants to merge 320 commits intohuggingface:mainfrom lapp0:policy_trainer_base

+1,088

This pull request is big! We're only showing the most recent 250 commits

Commits on Apr 24, 2024

remove micro_batch_inds
Andrew Lapp
committed
Try putting on same device
Andrew Lapp
committed
prepare ref model
Andrew Lapp
committed
clean up ref model with context manager
Andrew Lapp
committed
clean up ref model with context manager
Andrew Lapp
committed
fix token names
Andrew Lapp
committed
fix ref model preparation
Andrew Lapp
committed

Commits on Apr 25, 2024

split into policy trainer base and rloo trainer
Andrew Lapp
committed
include PolicyTrainerBase in __init__.py
Andrew Lapp
committed
fix policy trainer base signature
Andrew Lapp
committed
fix chess typo
Andrew Lapp
committed
need specific generation config in def generate()
Andrew Lapp
committed
need specific generation config in def generate()
Andrew Lapp
committed
debug log
Andrew Lapp
committed
debug log
Andrew Lapp
committed
debug log
Andrew Lapp
committed
fix syntax
Andrew Lapp
committed
debug log
Andrew Lapp
committed
try this
Andrew Lapp
committed
try this
Andrew Lapp
committed
fix ref_model_mgr
Andrew Lapp
committed
fix eos token
Andrew Lapp
committed
try fix eos token
Andrew Lapp
committed
debug log
Andrew Lapp
committed
debug log
Andrew Lapp
committed
debug log
Andrew Lapp
committed
debug log
Andrew Lapp
committed
debug log
Andrew Lapp
committed
debug log
Andrew Lapp
committed
debug log
Andrew Lapp
committed
fix optimizer reference
Andrew Lapp
committed
fix logprobs
Andrew Lapp
committed
fix logprobs
Andrew Lapp
committed
fix log
Andrew Lapp
committed
fix log
Andrew Lapp
committed
remove approxkl
Andrew Lapp
committed
fix log stats
Andrew Lapp
committed
fix log stats
Andrew Lapp
committed
fix log stats
Andrew Lapp
committed
fix log stats
Andrew Lapp
committed
remove prints
Andrew Lapp
committed
clean up, don't do optimizer step
Andrew Lapp
committed
replicate refactor from main branch
Andrew Lapp
committed
debug
Andrew Lapp
committed
cleanup
Andrew Lapp
committed
remove debug log
Andrew Lapp
committed
add PolicyTrainerArguments, and incomplete ppov2
Andrew Lapp
committed
readd first_true_indices
Andrew Lapp
committed
readd reward model
Andrew Lapp
committed
more work on ppov2
Andrew Lapp
committed

Commits on Apr 26, 2024

update params
Andrew Lapp
committed
prepare reward model on accelerator
Andrew Lapp
committed
fix multigpu
Andrew Lapp
committed
neuter acceleration of quantized
Andrew Lapp
committed
try cleaning cache
Andrew Lapp
committed
re-add dropout disabling
Andrew Lapp
committed
fix peft check for unsloth
Andrew Lapp
committed
add debug log
Andrew Lapp
committed
try fixing peft model
Andrew Lapp
committed
fix
Andrew Lapp
committed
fix
Andrew Lapp
committed
fix
Andrew Lapp
committed
check if PeftModel
Andrew Lapp
committed
todos, and no .pretrained_Model
Andrew Lapp
committed
fix decorator
Andrew Lapp
committed
fix decorator
Andrew Lapp
committed
fix decorator, return model with disabled adapters if peft model
Andrew Lapp
committed
remove mutation of reward_model
Andrew Lapp
committed

Commits on Apr 27, 2024

try fixing casting
Andrew Lapp
committed
fix trailing comma
Andrew Lapp
committed
fix peft casting
Andrew Lapp
committed
fix peft casting
Andrew Lapp
committed
add eval_mode
Andrew Lapp
committed
add eval_mode
Andrew Lapp
committed
fix typo
Andrew Lapp
committed
fix typo
Andrew Lapp
committed
fix typo
Andrew Lapp
committed
fix typo
Andrew Lapp
committed
fix typo
Andrew Lapp
committed
fix typo
Andrew Lapp
committed
fix typo
Andrew Lapp
committed
try torch.no_grad
Andrew Lapp
committed
try torch.no_grad in other context
Andrew Lapp
committed
try not decorating backward
Andrew Lapp
committed
try hack
Andrew Lapp
committed
remove hack
Andrew Lapp
committed
set output.requires_grad_(True) hook, re-add self._cast_base_model_ctx() to accelerator.backwards
Andrew Lapp
committed
follow dpo compute_loss() paradigm
Andrew Lapp
committed
don't backprop in compute loss
Andrew Lapp
committed
bug fix: I was always calling no_grad, even when running forward pass for loss calculation in model
Andrew Lapp
committed
keep stored metrics
Andrew Lapp
committed
fix import error
Andrew Lapp
committed

Commits on Apr 28, 2024

fix typo
Andrew Lapp
committed
try disabling eval_mode
Andrew Lapp
committed
add print statement
Andrew Lapp
committed
cast metrics to float to allow averaging
Andrew Lapp
committed
ensure autocast
Andrew Lapp
committed
ensure autocast
Andrew Lapp
committed
add nan detection debug log
Andrew Lapp
committed
remove pre-forward check
Andrew Lapp
committed
fix detect grad
Andrew Lapp
committed
try accumulating to get grad
Andrew Lapp
committed
add assertions to ensure grad can be calculated
Andrew Lapp
committed
debug log
Andrew Lapp
committed
remove portion which is probably breaking computation graph
Andrew Lapp
committed
new logging
Andrew Lapp
committed
log grad_fn
Andrew Lapp
committed
set breakpoint
Andrew Lapp
committed
remove debug break and logs
Andrew Lapp
committed

Commits on Apr 29, 2024

try saving memory by using amp dtype instead of float32
Andrew Lapp
committed
force bfloat casting
Andrew Lapp
committed
see if torch cuda amp autocast saves memory
Andrew Lapp
committed
try forcing bfloat16
Andrew Lapp
committed
comment out probably unnecessary code
Andrew Lapp
committed
remove print statement
Andrew Lapp
committed
add tag labelling
Andrew Lapp
committed
allow model initialization within trainer by adapting dpo
Andrew Lapp
committed
merge
Andrew Lapp
committed
revert un-adapted files
Andrew Lapp
committed
revert un-adapted files
Andrew Lapp
committed
revert un-adapted files
Andrew Lapp
committed

Commits on Apr 30, 2024

partial work refactoring ppov2 for replication w/ peft
Andrew Lapp
committed
add timer to metrics
Andrew Lapp
committed
revert moving
Andrew Lapp
committed
revert ppo.py
Andrew Lapp
committed

Commits on May 1, 2024

fix syntax error
Andrew Lapp
committed
fix syntax error
Andrew Lapp
committed
fix syntax error
Andrew Lapp
committed
fix syntax error
Andrew Lapp
committed
fix bad signature
Andrew Lapp
committed
apply fast eval mode
Andrew Lapp
committed
fix fast_eval_mode
Andrew Lapp
committed
use no_grad, not inference_mode
Andrew Lapp
committed
fix fast_eval_model w/ unsloth
Andrew Lapp
committed

Commits on May 2, 2024

don't use FastLanguageModel.for_inference
Andrew Lapp
committed
remove experiment that only works with llama models
Andrew Lapp
committed
try putting it all in autocast, maybe that helps with nan grad?
Andrew Lapp
committed
try replicating dpo autocast
Andrew Lapp
committed
try replicating dpo generate
Andrew Lapp
committed
am I even supposed to cast?
Andrew Lapp
committed
revert
Andrew Lapp
committed
revert
Andrew Lapp
committed
try fix
Andrew Lapp
committed
try fix
Andrew Lapp
committed
add backward pass log hook
Andrew Lapp
committed
add backward pass log hook, fix
Andrew Lapp
committed
add backward pass log hook, fix
Andrew Lapp
committed
add backward pass log hook, fix
Andrew Lapp
committed
add backward pass log hook, fix
Andrew Lapp
committed
add backward pass log hook, fix
Andrew Lapp
committed
add backward pass log hook, fix
Andrew Lapp
committed
add backward pass log hook, fix
Andrew Lapp
committed
add backward pass log hook, fix
Andrew Lapp
committed
add backward pass log hook, fix
Andrew Lapp
committed
add backward pass log hook, fix
Andrew Lapp
committed
add backward pass log hook, fix
Andrew Lapp
committed
remove debug logging, found the step where backprop fails
Andrew Lapp
committed
try fixing ref_model
Andrew Lapp
committed
try fixing ref_model ctx teardown
Andrew Lapp
committed
try skipping addition of invalid logprobs
Andrew Lapp
committed
try removing anything that might by chance interfere with backprop
Andrew Lapp
committed
try removing all INVALID_LOGPROB
Andrew Lapp
committed
revert: nope, it's not unsloth accidentally ignoring attention mask for INVALID_LOGPROB
Andrew Lapp
committed
debug print statements
Andrew Lapp
committed
fix typo
Andrew Lapp
committed
fix typo
Andrew Lapp
committed
break to example logprobs
Andrew Lapp
committed
change default temp, update gather index
Andrew Lapp
committed
remove breakpoint
Andrew Lapp
committed
cleanup softmax+gather, add break
Andrew Lapp
committed
fix err
Andrew Lapp
committed
use true temp
Andrew Lapp
committed
remove break
Andrew Lapp
committed
add breakpoint
Andrew Lapp
committed
hacky fix
Andrew Lapp
committed
add break
Andrew Lapp
committed
try output logits, not scores?
Andrew Lapp
committed
bugfix
Andrew Lapp
committed

Commits on May 3, 2024

separate generate and forward
Andrew Lapp
committed
fix conditional
Andrew Lapp
committed
try not casting gradients
Andrew Lapp
committed
cast locally
Andrew Lapp
committed
fix syntax
Andrew Lapp
committed
update logging
Andrew Lapp
committed
try multiplying loss?
Andrew Lapp
committed
scale loss by less
Andrew Lapp
committed
log advantages, and new advantage method
Andrew Lapp
committed
fix mistake in formula
Andrew Lapp
committed
use new advantage formula
Andrew Lapp
committed
fix advantage formula
Andrew Lapp
committed
fix advantage formula
Andrew Lapp
committed

Commits on May 4, 2024

refactoring experiment
Andrew Lapp
committed

Commits on May 5, 2024

fix syntax error
Andrew Lapp
committed
fix signature
Andrew Lapp
committed
fix signature
Andrew Lapp
committed
fix context manager
Andrew Lapp
committed
add decorator
Andrew Lapp
committed
prototyping update model
Andrew Lapp
committed
fix import
Andrew Lapp
committed
fix import
Andrew Lapp
committed
remove from utils
Andrew Lapp
committed
fix typo
Andrew Lapp
committed
fix inconsistent var name
Andrew Lapp
committed
try using dynamicdataloader
Andrew Lapp
committed
fix signature
Andrew Lapp
committed
fix signature
Andrew Lapp
committed
fix dynamic data loader
Andrew Lapp
committed
fix syntax
Andrew Lapp
committed
fix name error
Andrew Lapp
committed
fix name error, add tqdm to dynamic data loader
Andrew Lapp
committed
fix import error
Andrew Lapp
committed
clean cache help with memory
Andrew Lapp
committed
clean cache help with memory
Andrew Lapp
committed
try improving memory usage
Andrew Lapp
committed
fix mutate_fn
Andrew Lapp
committed

Commits on May 6, 2024

fix call to first_true_indices
Andrew Lapp
committed
fix missing var
Andrew Lapp
committed
syntax error
Andrew Lapp
committed
log problematic key
Andrew Lapp
committed
fix entropy logging
Andrew Lapp
committed
remove logging prints, try to fix ref model mgr
Andrew Lapp
committed
fix typo
Andrew Lapp
committed
fix set adapter
Andrew Lapp
committed
try disable cache on forward
Andrew Lapp
committed
try disabling cache the entire train run
Andrew Lapp
committed
try fixing unsloth nan/inf by casting throughout loss calculation
Andrew Lapp
committed
cleanup
Andrew Lapp
committed
fix import error
Andrew Lapp
committed
separate get_batch_loss_metrics
Andrew Lapp
committed
remove timer, ensure forward doesn't use cache
Andrew Lapp
committed
cleanup, cast during creation of extras
Andrew Lapp
committed
try temporary hack
Andrew Lapp
committed
try this instead
Andrew Lapp
committed
revert utils casting change
Andrew Lapp
committed
disable cache in generate
Andrew Lapp
committed
cleanup
Andrew Lapp
committed
fix autocast
Andrew Lapp
committed
try fixing generation config
Andrew Lapp
committed
cleanup
Andrew Lapp
committed
cleanup
Andrew Lapp
committed
prototype ppov2
Andrew Lapp
committed

Commits on May 7, 2024

cleanup, disable cache in forward
Andrew Lapp
committed
fix import
Andrew Lapp
committed
add logging
Andrew Lapp
committed
add logging
Andrew Lapp
committed
try maintaining disable_adapter context
Andrew Lapp
committed
try fixing ref model mgr
Andrew Lapp
committed
try fixing ref model mgr
Andrew Lapp
committed
try fixing ref model mgr
Andrew Lapp
committed
remove logs, the manager is fixed
Andrew Lapp
committed