Sync with torchtitan#2

Closed

philippguevorguian wants to merge 263 commits intomainfrom sync_with_torchtitan

+14-21

This pull request is big! We're only showing the most recent 250 commits

Commits on Feb 24, 2024

Commits on Feb 26, 2024

add iter time tracking via cuda events, add data loading times, add columnar display to show both, show avg iter & data loading times at end of training (pytorch#87 )
lessw2020
authored
Fill missing options in toml file wih argparse defaults (pytorch#91 )

gnadathur
and
gnadathur
authored

Commits on Feb 28, 2024

Enable libUV for torchtrain (pytorch#98 )

gnadathur
and
gnadathur
authored

Commits on Feb 29, 2024

Commits on Mar 1, 2024

Add job description field in toml (pytorch#101 )

gnadathur
and
gnadathur
authored

Commits on Mar 2, 2024

fix 2D parallel crash caused by all-reduce on 2D world_mesh
tianyu-l
committed

Commits on Mar 5, 2024

Load missing keys default from argparse (pytorch#111 )

gnadathur
and
gnadathur
authored
Add meta_init, enable it as default init process (pytorch#84 )
lessw2020
authored
Fix feedback from PR 111 (pytorch#113 )

gnadathur
and
gnadathur
authored

Commits on Mar 6, 2024

Commits on Mar 7, 2024

Commits on Mar 12, 2024

Commits on Mar 13, 2024

Commits on Mar 15, 2024

Commits on Mar 21, 2024

disable buffer reuse for compile for now (pytorch#156 )
wanchaol
authored

Commits on Mar 22, 2024

refactor config manager and support cmd overrides (pytorch#157 )
wanchaol
authored

Commits on Mar 24, 2024

Add support for generating debug traces on failure
chauhang
authored

Commits on Mar 25, 2024

rename sequence_parallel to tensor_parallel (pytorch#162 )
wanchaol
authored

Commits on Mar 28, 2024

Commits on Mar 29, 2024

Removed setting global flag for swap_tensors since not needed anymore
awgu
committed

Commits on Apr 2, 2024

Add integration test with compile enabled (pytorch#183 )

gnadathur
and
gnadathur
authored

Commits on Apr 3, 2024

remove folding and unfolding of sequence dim in model.py
tianyu-l
committed

Commits on Apr 4, 2024

bump comm.train_timeout_seconds (pytorch#189 )
wanchaol
authored

Commits on Apr 5, 2024

Commits on Apr 10, 2024

Commits on Apr 11, 2024

Commits on Apr 15, 2024

codebase cleanup
tianyu-l
committed

Commits on Apr 16, 2024

Commits on Apr 17, 2024

Commits on Apr 20, 2024

Support Llama3 8b/70b (pytorch#256 )
wanchaol
authored

Commits on Apr 22, 2024

polish llama 3 setup
tianyu-l
committed

Commits on Apr 23, 2024

reenable integration tests with a test tokenizer (pytorch#259 )
wanchaol
authored

Commits on Apr 29, 2024

Commits on Apr 30, 2024

Commits on May 1, 2024

Add periodic integration test with signal (pytorch#289 )
gnadathur
authored

Commits on May 3, 2024

Commits on May 7, 2024

Implement async_checkpoint
fegin
committed

Commits on May 8, 2024

simplify embedding + first transformer block TP (pytorch#314 )
wanchaol
authored

Commits on May 10, 2024

Only include checkpoints that have .metadata written (pytorch#315 )
liangluofb
authored

Commits on May 13, 2024

Commits on May 15, 2024

Commits on May 16, 2024

Add a workflow to build torchtitan-ubuntu-20.04-clang12 Docker image for CI (pytorch#338 )
huydhn
authored

Commits on May 17, 2024

Commits on May 21, 2024

Commits on May 23, 2024

Update pipelining import after change on pytorch
wconstab
committed

Commits on May 24, 2024

Commits on May 29, 2024

Commits on May 30, 2024

Commits on May 31, 2024

keep only latest k checkpoints (pytorch#366 )
liangluofb
authored

Commits on Jun 3, 2024

Commits on Jun 4, 2024

Make metrics logging work for pipeline parallelism
wconstab
committed

Commits on Jun 5, 2024

[RFC] Allow ModelWrapper and OptimizerWrapper to accept multiple models

fegin
authored and
wconstab
committed
Add 3D support
wconstab
committed

Commits on Jun 6, 2024

Commits on Jun 7, 2024

Abstract out out optimizer params and update foreach calling convention (pytorch#386 )
drisspg
authored

Commits on Jun 9, 2024

Commits on Jun 10, 2024

Commits on Jun 11, 2024

Fix 1D PP tracer test

kwen2501
authored and
wconstab
committed

Commits on Jun 14, 2024

Commits on Jun 17, 2024

Change debugmodel to have 8 layers
wconstab
committed

Commits on Jun 18, 2024

Prepare train.py for model chunks for pipelining
wconstab
committed

Commits on Jun 19, 2024

dump memory snapshot to analyze OOMs (pytorch#395 )
weifengpy
authored

Commits on Jun 20, 2024

whole_model for fp8 (pytorch#414 )
weifengpy
authored

Commits on Jun 21, 2024

Add train loop support for looped PP schedules
wconstab
committed

Commits on Jun 26, 2024

by default disable heavy memory profiling
tianyu-l
committed

Commits on Jun 27, 2024

Add the option to turn on async-TP
yifuwang
committed

Commits on Jul 1, 2024

Modifying memory estimation options and minor changes
sanketpurandare
committed

Commits on Jul 8, 2024

Commits on Jul 11, 2024

Commits on Jul 16, 2024

enable FSDP2 + fp8 all-gather and fix TP fp8 all-gather (pytorch#413 )
weifengpy
authored

Commits on Jul 18, 2024

Add support of DDP and experimental CompiledAutograd
fegin
committed

Commits on Jul 19, 2024

Commits on Jul 20, 2024

remove CI for FSDP2 + fp8 all-gather (pytorch#470 )
weifengpy
authored

Commits on Jul 21, 2024

dynamically update torch.compile cache config to ensure async tp support, enhance async tp UX (pytorch#471 )
lessw2020
authored

Commits on Jul 26, 2024

Fix 8gpu PP failure due to 2D DCP disablement
wconstab
committed
update float8 integration after UX changes (pytorch#484 )
vkuzo
authored
Re-enable FSDP2 Mem Tracker integration tests
Sanket Jayant Purandare
committed

Commits on Jul 29, 2024

Used partial instead of global vars for LR scheduling
awgu
committed

Commits on Aug 2, 2024

[TP] Infer local n_heads instead of ad-hoc model changes
kwen2501
committed

Commits on Aug 3, 2024

some compile-related updates
tianyu-l
committed

Commits on Aug 5, 2024

Commits on Aug 6, 2024

Add warning to compile rmsnorm (pytorch#505 )
wanchaol
authored

Commits on Aug 7, 2024

Commits on Aug 8, 2024

Commits on Aug 9, 2024

Commits on Aug 12, 2024

Commits on Aug 13, 2024

Throw warning if users are using old pytorch version that not including DTensor strided sharding (pytorch#507 )
XilunWu
authored

Commits on Aug 14, 2024

Update fsdp.md (pytorch#519 )
crcrpar
authored

Commits on Aug 15, 2024

remove old torch dependency in requirements.txt
tianyu-l
committed

Commits on Aug 16, 2024

Commits on Aug 20, 2024

Merge branch 'main' into sync_with_torchtitan
philippguevorguian
authored