[CPU] Improve tile sizes selection for tensor.pack ops.

[hanhanW]

It disables special vector sizes for non-f32 cases because the logic is only for 16x16 transpose cases. The improvements of dispatch sizes are from vectorization. We are not able to vectorize named ops if they have dynamic shapes, which is fixed by llvm/llvm-project@03529b9. The change allows backends to vectorize them because they become static shapes (by tiling with size=1). It is not a hard condition; we track it in #15441

The revision takes the number of threads into account, so we have better performance on multi-threaded. It also reduces runtime overheads.

This is a step toward to #15391 and #15349

It improves the performance of tensor.pack op from 420 ms to 170 ms on 8-threaded x86 CPU.

[github-actions]

Abbreviated Benchmark Summary

@ commit 24fc0109d548e8dff4a7cecb75ece805f5920356 (vs. base a4b1a78646f65e8e84a73fcf547bb75a60af5cf1)

Regressed Latencies 🚩

Benchmark Name	Average Latency (ms)	Median Latency (ms)	Latency Standard Deviation (ms)
MobileNetV2\_fp32(tflite) [x86\_64-cascadelake-linux\_gnu-llvm\_cpu][default-flags] local\_task(embedded\_elf)[1-thread,full-inference,default-flags] with default @ c2-standard-16[cpu]	15.620 (vs. 12.891, 21.16%↑)	15.634	0.081
Falcon7bGptqPT(linalg) [x86\_64-cascadelake-linux\_gnu-llvm\_cpu][default-flags] local\_task(embedded\_elf)[8-thread,full-inference,default-flags] with default @ c2-standard-16[cpu]	45014.478 (vs. 37280.459, 20.75%↑)	45013.931	362.379
DeepLabV3\_fp32(tflite) [x86\_64-cascadelake-linux\_gnu-llvm\_cpu][default-flags] local\_task(embedded\_elf)[1-thread,full-inference,default-flags] with default @ c2-standard-16[cpu]	38.889 (vs. 33.125, 17.40%↑)	38.817	0.347

[Top 3 out of 10 results showed]

Regressed Total Dispatch Sizes 🚩

Benchmark Name	Total Dispatch Size (bytes)
MobileBertSquad\_fp32(tflite) [armv8.2-a-generic-linux\_android29-llvm\_cpu][experimental-flags,data-tiling,ukernel,compile-stats]	26984 (vs. 25512, 5.77%↑)

Improved Total Dispatch Sizes 🎉

Benchmark Name	Total Dispatch Size (bytes)
Vit\_int8(tflite) [x86\_64-cascadelake-linux\_gnu-llvm\_cpu][experimental-flags,data-tiling,ukernel,compile-stats]	766072 (vs. 1366184, 43.93%↓)
PersonDetect\_int8(tflite) [x86\_64-cascadelake-linux\_gnu-llvm\_cpu][experimental-flags,data-tiling,ukernel,compile-stats]	65256 (vs. 106056, 38.47%↓)
Falcon7bGptqPT(linalg) [x86\_64-cascadelake-linux\_gnu-llvm\_cpu][experimental-flags,data-tiling,ukernel,compile-stats]	103688 (vs. 131144, 20.94%↓)

[Top 3 out of 6 results showed]

Improved Total Artifact Sizes 🎉

Benchmark Name	Total Artifact Size (bytes)
PersonDetect\_int8(tflite) [x86\_64-cascadelake-linux\_gnu-llvm\_cpu][experimental-flags,data-tiling,ukernel,compile-stats]	331845 (vs. 372613, 10.94%↓)
Falcon7bGptqPT(linalg) [x86\_64-cascadelake-linux\_gnu-llvm\_cpu][experimental-flags,data-tiling,ukernel,compile-stats]	344296 (vs. 371752, 7.39%↓)

For more information:

• Full benchmark result tables
• Build

Source Workflow Run

[MaheshRavishankar]

Looks fine to me... I dont have much of a handle on the heuristics here... If you have any data as to why it helps, post it here for recording.

[hanhanW]

Looks fine to me... I dont have much of a handle on the heuristics here... If you have any data as to why it helps, post it here for recording.

done. I include the perf number in the description.

[dcaballe]

Do you know if the Falcon7B regression is real? That's one of the main models we are working on right now.

[hanhanW]

Do you know if the Falcon7B regression is real? That's one of the main models we are working on right now.

I can take a look, but I feel that it is not real.. The regression did not happen before I rebase it to ToT. https://gist.github.com/iree-github-actions-bot/99c8f439051a13ad1a3d0ebe112056b9

[hanhanW]

I'm pretty sure that it is not real because it is on default flags, not data-tiling.