Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
TristanThrush authored Oct 21, 2024
1 parent 04ea7db commit 8e5ddc3
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,13 @@ Note that you don't need to run all of the scripts in these examples with a CUDA
but some of the pip packages (like `flash-attn`) will fail to install unless you are installing
on a device with `nvcc`. Then you can switch to another device. Most of the requirements here are for the plethora of random Hugging Face models from which you might want to get bits-per-byte (BPB) values.

Navigate to `get_error_and_bpb` for examples of getting BPB values and evaluation scores.
Navigate to `get_error_and_bpb/` for examples of getting BPB values and evaluation scores.
The examples compute BPB from different groups of ~100 open-source language models on a sample RedPajama V2
and also a synthetic pretraining dataset. They also compute evals on many different benchmarks using the
Eleuther Eval Harness.

Navigate to `get_fasttext_filter` for examples of using
the information from `get_error_and_bpb` to estimate good pretraining
Navigate to `get_fasttext_filter/` for examples of using
the information from `get_error_and_bpb/` to estimate good pretraining
sampling distributions. The examples save reusable fastText training data filters,
which you can plug into your LLM training pipeline, whatever it may be.

Expand Down

0 comments on commit 8e5ddc3

Please sign in to comment.