Skip to content

Commit

Permalink
docs: update readme
Browse files Browse the repository at this point in the history
  • Loading branch information
joennlae committed Nov 13, 2023
1 parent c4c0dcb commit f603e16
Showing 1 changed file with 12 additions and 6 deletions.
18 changes: 12 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,31 +1,37 @@

# Tensorli

Absolute minimalistic implementation of a GPT-like transformer using only numpy.
Absolute minimalistic implementation of a GPT-like transformer using only `numpy`.

The implementation includes:
- Automatic differentiation
- `Tensorli` object (PyTorch like)
- Simple NN layers: `Linearli`, `Embeddingli`, `MultiheadAttentionli`, `LayerNorm`
- Optimizers: `Adamli`

everything that is needed to train and execute a GPT-like transformer model.
All that is "needed" to train and execute a GPT-like transformer model.

```
Everything else are optimizations.
Everything else is optimizations.
```

## Naming

In the region where I grew up we have a word for "little" that is used as a suffix. For example, "little dog" would be "dogli". I thought it would be a nice name for a minimalistic implementation of a neural network library.
In the region where I grew up, a word for "little" is used as a suffix \[[1](https://de.wikipedia.org/wiki/-li)\]. For example, "little dog" would be "dogli". I thought it would be a nice name for a minimalistic implementation of a neural network library.

### Caveats

This library works but it is not optimized. It is not meant to be used in production. It is meant to be used as a learning tool.
This library works, but it is NOT optimized. It is not meant to be used in production or for anything at scale. It is meant to be used as a learning tool.

## Inspiration

This library is heavily inspired by the following projects:
- [minGPT](https://github.com/karpathy/minGPT)
- [tinygrad](https://github.com/tinygrad/tinygrad)

I highly recommend checking them out.

### Plans & Outlook

- [ ] Add Dropout
- [ ] Add more experimental architectures

0 comments on commit f603e16

Please sign in to comment.