load bf16 directly, and some "quality of life" handling of fp32/fp16/bf16 precisions#265

Merged

karpathy merged 12 commits intomasterfrom feature/load_bf16

Apr 28, 2024

+244-170

Commits on Apr 27, 2024

code to load bf16 weights directly, and also re-wire the position of tensors to put the layernorms at the end. the training loop seems to work ok, and the tests pass and the loss and optimization l…
karpathy
committed
i think i am making things cleaner, but i am not fixing the problem
karpathy
committed
i think github copilot betrayed me on this index here, i cant remember
karpathy
committed
fix dumb bug. i'll blame github copilot but i can't remember
karpathy
committed
tweak the tolerances until we pass lol
karpathy
committed
print more in the comparison
karpathy
committed
fix a really bad bug in how i was checking the gradients, where i loaded them in the old order, so yeah...
karpathy
committed