dhcode-cpp / cut-cross-entropy-pytorch Public

Notifications You must be signed in to change notification settings
Fork 0
Star 5

pytorch notebook for implemention for cut-cross-entropy LLM training.

5 stars 0 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 8 Commits
README.assets		README.assets
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
cut_cross_entropy.ipynb		cut_cross_entropy.ipynb

Repository files navigation

✂️Cut-Cross-Entropy-Pytorch

It’s Pytorch-version Cut-Cross-Entropy(CCE) implementation:

GEMM
LSE-style CCE Forward
LSE CCE Backward
Linear-Cross-Entropy backward

Here is a blog for explaination of CCE: zhihu

Reference

Cut Your Losses in Large-Vocabulary Language Models

About

pytorch notebook for implemention for cut-cross-entropy LLM training.

training ai ml distributed llm

Report repository

Releases

No releases published

Packages

No packages published

Languages

Jupyter Notebook 100.0%