llava: batch inference

[Borobo]

Hello!

I'm using llava with the server and I'm wondering if anyone is working on batch inference by batching llava's clip or not. If not, I would be happy to contribute as this feature could be very useful to speed up inference time for llava.
As i'm new in this wonderful repo, I would also be grateful if you could guide me for the first steps (what file do I have to edit)

Thanks you in advance!

[cmp-nct]

Afaik FSSRepo was working on offloading CLIP properly to GPU, there is a PR for that already but it was missing some ggml cuda upgrades (broadcasting features I think).

Currently it runs on CPU only which is a massive slowdown, I don't think parallel processing would help in any way on CPU. But of course on GPU it's a different story.

[Borobo]

So I guess I'll have to wait until this PR is merged before considering clip batching functionality, as I'm not very familiar with ggml structures. I'm going to keep an eye on it, thanks for you reply!

[y10ab1]

We can track the PR ggerganov/llama.cpp/pull/4205.
It appears that the corresponding pull request at ggerganov/ggml/pull/621 has been merged.
I guess there may be progress soon.

[KohakuBlueleaf]

Is there any updates for this feature?

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.