[WIP] Faster generation with CUDA graphs#65

Merged

tsengalb99 merged 7 commits intomainfrom faster_generate

Aug 28, 2024

+2,326-666

Commits on Jul 21, 2024

backup
Albert Tseng
committed
working?
Albert Tseng
committed
wow this actually works
Albert Tseng
committed

Commits on Jul 22, 2024

even faster
Albert Tseng
committed
4 bit too
Albert Tseng
committed

Commits on Aug 6, 2024

time decode only
Ubuntu
committed

Commits on Aug 28, 2024

patch pos id
Albert Tseng
committed