update readme.md and performance.md #258
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[ghstack-poisoned]
tianyu-l
added a commit
that referenced
this pull request
Apr 22, 2024
ghstack-source-id: 1a616ac8fa3b3060ca568d04dff20626259dcd8e Pull Request resolved: #258
facebook-github-bot
added
the
CLA Signed
Include llama3 performance metrics. [ghstack-poisoned]
tianyu-l
added a commit
that referenced
this pull request
Apr 22, 2024
ghstack-source-id: 7c6c6898f662142c29a535766e6efa8e0e13d61f Pull Request resolved: #258
wanchaol
reviewed
Apr 22, 2024
docs/performance.md
Outdated
| @@ -1,23 +1,43 @@ | |||
| To demonstrate the effectiveness of techniques used in the torchtitan, we report both the infra metrics and loss curves of the LLaMa 13B and the LLaMa 70B training on 64 A100 (80GB memory) GPUs. We report infra metrics achieved by FSDP2 (1D parallelism) under various configurations, and loss curves for both 1D parallelism (FSDP2) and 2D parallelism (FSDP2 + Tensor Parallel) training. | |||
| To demonstrate the effectiveness of techniques used in torchtitan, we report both the infra metrics and loss curves of LLaMa 2 (13B and 70B) and LLaMa 3 (8B and 70B) training on 64 A100 (80GB memory) GPUs. We report infra metrics achieved by FSDP2 (1D parallelism) under various configurations, and loss curves for both 1D parallelism (FSDP2) and 2D parallelism (FSDP2 + Tensor Parallel) training. | |||
There was a problem hiding this comment.
For FSDP2, I think it worth linking to the FSDP2.readme
docs/performance.md
Outdated
| | 70B | 2 | full | 353 | 50.5% | | ||
|
|
||
| We mostly use local batch size 2 (global batch size = local batch size 2 * number of FSDP ranks 64 = 128) in the experiments, because it mimics the small local batch size in large scaled training, and moreoever allows us to compare 1D (FSDP) and 2D (FSDP + TP) training under the same global batch size on both 13B and 70B LLaMa models, without the out-of-memory (OOM) issue. In fact, for the 70B model with full activation checkpointing, the MFU can go up to 54% when local batch size is higher (but before OOM happens). | ||
| We mostly use local batch size 2 (global batch size 128) in the experiments, to keep the same number of tokens per training iteration between LLaMa 2 and LLaMa 3 (since the default sequence length in LLaMa 2 is 4096 which is halved compared with LLaMa 3). In fact, for LLaMa 2 70B model with full activation checkpointing, the MFU can go up to 54% when local batch size is higher (but before OOM happens). |
There was a problem hiding this comment.
"LLaMa 2/3" -> "Llama 2/3"
docs/performance.md
Outdated
| We mostly use local batch size 2 (global batch size = local batch size 2 * number of FSDP ranks 64 = 128) in the experiments, because it mimics the small local batch size in large scaled training, and moreoever allows us to compare 1D (FSDP) and 2D (FSDP + TP) training under the same global batch size on both 13B and 70B LLaMa models, without the out-of-memory (OOM) issue. In fact, for the 70B model with full activation checkpointing, the MFU can go up to 54% when local batch size is higher (but before OOM happens). | ||
| We mostly use local batch size 2 (global batch size 128) in the experiments, to keep the same number of tokens per training iteration between LLaMa 2 and LLaMa 3 (since the default sequence length in LLaMa 2 is 4096 which is halved compared with LLaMa 3). In fact, for LLaMa 2 70B model with full activation checkpointing, the MFU can go up to 54% when local batch size is higher (but before OOM happens). | ||
|
|
||
| Next we show the loss curves for LLaMa 2 13B and LLaMa 2 70B training with both 1D parallelism (FSDP2) and 2D parallelism (FSDP2 + Tensor Parallel). All the four models are trained 3000 steps with global batch size 128. In terms of activation checkpointing (AC) configs, the LLaMa 2 13B training jobs use selective op AC, whereas the LLaMa 70B training jobs use full AC. The results are shown in the picture (a TensorBoard screenshot) below[^3]. |
docs/performance.md
Outdated
|
|
||
|  | ||
| [^1]: We note that on 128 A100 GPUs, the MFU of LLaMa 3 70B training can go up to 50.9%. |
There was a problem hiding this comment.
This in fact is for Llama 3
awgu
reviewed
Apr 23, 2024
docs/performance.md
Outdated
| @@ -1,23 +1,43 @@ | |||
| To demonstrate the effectiveness of techniques used in the torchtitan, we report both the infra metrics and loss curves of the LLaMa 13B and the LLaMa 70B training on 64 A100 (80GB memory) GPUs. We report infra metrics achieved by FSDP2 (1D parallelism) under various configurations, and loss curves for both 1D parallelism (FSDP2) and 2D parallelism (FSDP2 + Tensor Parallel) training. | |||
| To demonstrate the effectiveness of techniques used in torchtitan, we report both the infra metrics and loss curves of LLaMa 2 (13B and 70B) and LLaMa 3 (8B and 70B) training on 64 A100 (80GB memory) GPUs. We report infra metrics achieved by FSDP2 (1D parallelism) under various configurations, and loss curves for both 1D parallelism (FSDP2) and 2D parallelism (FSDP2 + Tensor Parallel) training. | |||
There was a problem hiding this comment.
Should we change these "LLaMa" to "Llama"? (It could be worth a find and replace.)
awgu
reviewed
Apr 23, 2024
README.md
Outdated
| @@ -49,9 +49,9 @@ pip install -r requirements.txt | |||
|
|
|||
| ### Downloading a tokenizer.model | |||
|
|
|||
| `torchtitan` currently supports training Llama3 (8B, 70B), and Llama2 (13B, 70B) out of the box. To get started training these models, we need to download a tokenizer.model. Follow the instructions on the official [meta-llama](https://huggingface.co/meta-llama/Meta-Llama-3-8B) repository to ensure you have access to the Llama model weights. | |||
| `torchtitan` currently supports training Llama 3 (8B, 70B), and Llama 2 (13B, 70B) out of the box. To get started training these models, we need to download a tokenizer.model. Follow the instructions on the official [meta-llama](https://huggingface.co/meta-llama/Meta-Llama-3-8B) repository to ensure you have the access to Llama model weights. | |||
There was a problem hiding this comment.
nit: I think the original placement of "the" in "you have access to the Llama model weights" sounds more natural than the new placement.
Include llama3 performance metrics. [ghstack-poisoned]
tianyu-l
added a commit
that referenced
this pull request
Apr 23, 2024
ghstack-source-id: c6485a3119b85b9a81c04a68fa6070a3e62cd839 Pull Request resolved: #258
Include llama3 performance metrics. [ghstack-poisoned]
tianyu-l
added a commit
that referenced
this pull request
Apr 23, 2024
ghstack-source-id: dbf44a759c9544f5b46a2c71789132d64385ed77 Pull Request resolved: #258
wanchaol
reviewed
Apr 24, 2024
README.md
Outdated
| #### (4/18/2024): `torchtitan` is now public but in a pre-release state and under development. | ||
| Currently we showcase pre-training Llama2 models (LLMs) of various sizes from scratch. `torchtitan` is tested and verified with the PyTorch nightly version `torch-2.4.0.dev20240412`. (We recommend latest PyTorch nightly). | ||
| #### (4/23/2024): `torchtitan` is now public but in a pre-release state and under development. | ||
| Currently we showcase pre-training Llama 3 and Llama 2 models (LLMs) of various sizes from scratch. `torchtitan` is tested and verified with the PyTorch nightly version `torch-2.4.0.dev20240412`. (We recommend latest PyTorch nightly). |
There was a problem hiding this comment.
nit: maybe let's emphasize Llama 3 and Lllama 2
wanchaol
approved these changes
Apr 24, 2024
Include llama3 performance metrics. [ghstack-poisoned]
tianyu-l
added a commit
that referenced
this pull request
Apr 24, 2024
ghstack-source-id: 17aed01a620f1dabb65b4ff2944ad30a01dbb1e3 Pull Request resolved: #258
lessw2020
reviewed
Apr 24, 2024
lessw2020
reviewed
Apr 24, 2024
lessw2020
reviewed
Apr 24, 2024
lessw2020
reviewed
Apr 24, 2024
lessw2020
reviewed
Apr 24, 2024
lessw2020
reviewed
Apr 24, 2024
lessw2020
approved these changes
Apr 24, 2024
Include llama3 performance metrics. [ghstack-poisoned]
tianyu-l
added a commit
that referenced
this pull request
Apr 24, 2024
ghstack-source-id: dbf54574eae4c0d1447a40e1b4f65eb8ee46bff7 Pull Request resolved: #258
Include llama3 performance metrics. [ghstack-poisoned]
tianyu-l
added a commit
that referenced
this pull request
Apr 24, 2024
ghstack-source-id: a9bd1d33bf7bc9f5055a645c9639bcbe628afbfb Pull Request resolved: #258
tianyu-l
added a commit
that referenced
this pull request
Apr 24, 2024
ghstack-source-id: a9bd1d33bf7bc9f5055a645c9639bcbe628afbfb Pull Request resolved: #258
tianyu-l
added a commit
to tianyu-l/torchtitan_intern24
that referenced
this pull request
Aug 16, 2024
ghstack-source-id: a9bd1d33bf7bc9f5055a645c9639bcbe628afbfb Pull Request resolved: pytorch#258
philippguevorguian
pushed a commit
to YerevaNN/YNNtitan
that referenced
this pull request
Aug 17, 2024
ghstack-source-id: a9bd1d33bf7bc9f5055a645c9639bcbe628afbfb Pull Request resolved: pytorch#258
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack (oldest at bottom):
Include llama3 performance metrics.