Create llamacpp_HF loader

[oobabooga]

Still experimental.

Motivation

Having a wrapper similar to ExLlama_HF but for llama.cpp. The main goal is to evaluate llama.cpp perplexity and compare it to AutoGPTQ/ExLlama directly. As a bonus, it will allow the transformers samplers to be used for llama.cpp.

How to use it

1. Download oobabooga/llama-tokenizer using the UI or the download script:

python download-model.py oobabooga/llama-tokenizer

2. Load a ggml model with --loader llamacpp_HF or by setting the loader to llamacpp_HF in the UI.

Current status

Functional, but 3x slower:

llamacpp_HF:

Output generated in 15.79 seconds (12.60 tokens/s, 199 tokens, context 7, seed 313220017)

llamacpp:

Output generated in 5.50 seconds (36.39 tokens/s, 200 tokens, context 7, seed 1413621777)

[oobabooga]

The evaluation fails due to the following problem. The forward call

logits = torch.tensor(self.model.model.eval_logits).view(1, 1, -1).to(kwargs['input_ids'].device)

returns a tensor with shape torch.Size([1, 1, 32000]), while the evaluation code expects a tensor with shape torch.Size([1, 1200, 32000]), where the second number is whatever context size you chose.

I don't understand why the AutoGPTQ / transformers call returns a logit vector for each position in the input array, or how to reproduce that behavior. I'll tag @TheBloke because maybe he can help.