Metal: Support older iOS devices that not supported SIMD?

[jhen0409]

Currently working on using llama.cpp in iOS projects.

Although our main goal is to run on M1/M2. But started testing with my 2019 iPad Pro (A12Z). Unfortunately, I found that Metal cannot compile some kernel functions normally, the reason is that simd_sum is not supported in my device.

I found out that SIMD-scoped permute operations is indeed supported after A13 (apple6) in https://developer.apple.com/metal/Metal-Feature-Set-Tables.pdf.

Not an urgent problem, just curious if there is a way we can override this by a custom function. I'm very new to Metal.

[ggerganov]

I'm also new to Metal, so don't know the answer.
Can you try to #ifdef it somehow - if simd_sum is missing then do something else even if less efficient?

[jhen0409]

Thanks for comment!

At the moment it's not clear to me how to implement simd_sum correctly. Currently I just added an another check by use newLibraryWithSource to avoid crash.

It is certain that the metal backend uses simd_sum since 1cbf561, I'll try if it's possible to do the backward compatibility in an easy way. Maybe @lshzh-ww or @ikawrakow know something?

[lshzh-ww]

I believe the most practical approach would be using threadgroup memory to share the data between all the threads - same as ggerganov suggested. You have to add #ifdef in ggml_metal.metal file to do the reduce ops instead of using simd_sum. As you'll be leveraging threadgroup memory, it will also be essential to make modifications to the ggml_metal.m file, specifically to allocate and reserve threadgroup memory for the kernels.