BatchedExecutor LLava #770
Conversation
… support for Llava! - Switched batch queue in `BatchedExecutor` to have 2 possible types - token batches and embeddings batches. - Switched inference lock to a integer using Interlocked, this interacts better with async. - Added `Conversation.Prompt` method for `Span<float>` (raw embeddings) and `SafeLlavaImageEmbedHandle`. - Added `LLamaBatchEmbeddings`, equivalent to `LLamaBatch` for embeddings instead of for tokens.
|
@martindevans, The PR seems ok for me. Not related with this PR and more generally speaking is hard to me to have an "Executor" that doesn't implement the ILlamaExecutor interface. If seems to be a lower level of abstraction that ILlamaExecutor and more difficult to use for the final user. |
Rinne had the same issue too. It is a lower level of abstraction, intended as the new foundation that high level executors (replacements for the current ones) can be built on. Since you've both been confused by this, any ideas for better names? |
Added support for embeddings prompting to
BatchedExecutor, allowing support for Llava!BatchedExecutorto have 2 possible types - token batches and embeddings batches.Conversation.Promptmethod forSpan<float>(raw embeddings) andSafeLlavaImageEmbedHandle.LLamaBatchEmbeddings, equivalent toLLamaBatchfor embeddings instead of for tokens.