[Refactor] Brand New Scheduler

[hikettei]

Workload (Final)

• Check the kernel manually for batch_norm/layer_norm/feed_forward/rope/attention/softmax/chunk/Transformer (add tests if needed)
• Embedding+Embedding, Softmax+Softmax
• Fuse Chunk/Attention, Failing Case
• GPT2 No Segv
• GPT2 in CI
• almost there; so fix the schedule + fix the schedule cache and merge
• no-grad wont change the kernel (make an issue first)
• realize regardless of pause/backward (create a table in ctx)
• optimize the kernel for backward

Memo

• GPT2 Test in CI
• final: update group-mergeable-p
• Chunk=1, Attention=4
• Test: Matmul(GeLU(X), X) Matmul(X, GeLU(X)) as well as GeLU=Sin
• fix for jit; it is ok to realize save for backward across pause/backward
• RoPE JIT=1
• BatchNorm JIT=1
• Transformer
  • SERIALIZE=1 worked
  • SERIALIZE=0? -> no segv, it is working!
• ideally scheduler.lisp < 400 lines
• docs for pprint-graph
• move pprint-graph to caten/aasm
• break the graph first?
  • patch for fast dynamic shape first. (Load is scheduled multiple times)
  • and chase down to unfused views? (merge views)
  • LayerNorm = 1 Kernels
  • New Scheduler fuses multiple kernels too much; Loop Fission by ISL Scheduler is NEEDED!

Workload

• BatchNorm/LayerNorm/Softmax/... = 1 Kernels
• RoPE JIT=1 and 2 kernels, and pass the tests
• Fix for dynamic shape compilation (faster!)
• Update: Scalar <-> Scalar Pointer Mutation
• Update: Matmul(Matmul(..., ...))
• Remove old scheduler
• update the Lowerer
• Softmax (10)
• Refactor
• todo: automatically determine PARALLEL, PARALLEL in default

Lowerer Workload

• Update how node-reads/node-writes are used for the schedule-graph
• Update Scalar Pointer Manipulation
  • Fix for !rand '(a b)
  • as well as randn
• Refactor exprify.lisp
• Update Scalar Pointer Manipulation
• (!softmax `(10)) (Restart and create an additional loop)
• Transformer
• Transformer = 29 Kernel is MUST