[Refactor] Introduce Memory Optimization based on MIP

[hikettei]

Remained Tasks

• Dead Code Elimination
• Index Computation Simplification
• Load Latency Hiding Optimization

Changes

• Softsign
• GeLU
• HardSwish
• Fix In-place mutation
• (After fixing in-place mutation) Fold composed tan in a single expr
• tag v0.01
• Index simplification
• scalar value mutation
• Lexiographical Orderが間違ってるのか，ISLのScheduleの段階でFuseされてないKernelがReverseされてない？fuse_outermost_loopで並び替える (特に: Get non-fused Embedding worked)
• hide latency optimization; load複数回やらない
• zero_dim_tensor_elimination
• use :separate option to get Embedding worked

[hikettei]

• Implement MIP Problem based Memory Planner class in memory-planner.lisp
  • Constant: Peak_Memory_Size
• https://arxiv.org/pdf/2203.00448
• Reading: https://discuss.tvm.apache.org/t/discussion-alignment-memory-planning/9730
• https://arxiv.org/pdf/2003.02369
  • their focus is on really complicated network; may not be optimal in my use-case
• white paper https://arxiv.org/abs/2002.03794
• 実装の方針
  • 1. MIPはOutput/Inputだけに対して解く
  • 2. memory-planner.lispのrefactor
  • 3. tmpvarは事前になるべくfloat _acc_x に置き換える
  • 4. Solve MIP based on linear-programming and memory-plannerでID上書き

[hikettei]

prep research

• a white paper on dl compiler https://arxiv.org/abs/2002.03794
• Offline scheduling algorithms: https://arxiv.org/pdf/2003.02369
• https://arxiv.org/pdf/2203.00448
• similar discussions in tvm: https://discuss.tvm.apache.org/t/discussion-alignment-memory-planning/9730
• https://arxiv.org/pdf/2203.00448
• https://proceedings.mlsys.org/paper_files/paper/2021/file/5b47430e24a5a1f9fe21f0e8eb814131-Paper.pdf (really good to get the dynamic nn compilation)
• heuristic or MIP? -> MIP
• https://arxiv.org/html/2407.12117v1
• ⭐️ https://dl.acm.org/doi/pdf/10.5555/314500.315082

[hikettei]

IQ足りないからgreedyで