[EPIC][CPU] Enable predictable performance on mixed-types GEMM using data-tiling

[hanhanW]

This EPIC tracks all the related work.

Tasks related to core functionality

Tasks

• Form cast ops and mmt4d ops into the same dispatch. #15680
• [DT][CPU] Lift linalg.generic to linalg.batch_matmul + transpose + arith.ext* #15214
• [EPIC][CPU] int4 sub-type emulation #15613
• [DT][CPU] Data-tiled i4 matmuls on LLVM-CPU #14195
• [Util] Add a type interface for controlling hoisting into globals #15558
• Support i4 types in ConstEval #15681

Tasks related to performance improvements

• [GlobalOptimization] Quantized matmul reassociation changes for data-tiling enablement #15585
• [DT][CPU] ConstEval folding on quantized matmul with data-tiling #15554
• Rationalize mmt4d tile sizes #15651
• Improve LHS tensor.pack on non-f32 types for x86 #15441
• linalg.matmul_transpose_b is not data-tiled
• Need support for tensor.pack direct vectorization #15600
• [CPU][DT] i8.i8.i32 mmt4d codegen kernel is 10x slower than ukernels #15611

llama2 specific tasks

Tasks

• [CPU] Performance tracking for llama2 with data tiling + ukernels #15566
• [CPU] Slow copy coming from tensor.insert_slice with dynamic dims #15195
• Enable e2e integration tests

[hanhanW]

@Max191 I think you have some local patches and ideas that are required for mixed-types data-tiling work, could you add them to tasklist accordingly?

@bjacob please help update this if there are on-going/TODO tasks in your mind.

@MaheshRavishankar I created an epic to help us understand better what needs to be done for mixed-types data-tiling, and the work we've been working on.

Thank you all for all the awesome work!

[hanhanW]

For small tasks, adding a brief description to tasklist is good enough. For large tasks, it would be good if you can create an issue/epic. It's not necessary to do it now, but please help add a brief description. Thank you!