DeepSeek-R1 GGUF can't be loaded #3404

kbradsha · 2025-01-22T01:29:20Z

Bug Report

GPT4All downloads the DeepSeek-R1 library, but receive an error attempting to load the model for chat.

Steps to Reproduce

Navigate to Models
Select DeepSeek-R1 bartowski
Upon successful download the model can be loaded by ollama
GPT4All reports "Could not load model due to invalid model file for DeepSeek-R1-Distill-Qwen-14B-Q4_0.gguf"

Expected Behavior

GGuf file should load as with all of the others. Ollama loads the same file fine.

Your Environment

GPT4All version: v3.6.1
Operating System: Ubuntu 24.04
Chat model used (if applicable): DeepSeek-R1

I notice in the Chat Template settings an error is present:
Syntax error: 1:267: error: This feature has not been supported yet

dantrez · 2025-01-23T18:10:55Z

I am having the same issue. Version 3.6.1, Windows 10 Pro.

QohoZ · 2025-01-24T19:56:45Z

I am having the same issue. Version 3.7.0, Windows 11 Pro. I tried many models of several people to run DeepSeek-R1, but without success so far.

kbradsha · 2025-01-25T02:18:10Z

I suspect the chat template needs modification to accommodate the new output format.

adgu · 2025-01-25T09:53:01Z

Same here. W11H, v3.7.0

BrushAway · 2025-01-25T10:42:07Z

Good morning all,

Just to confirm, I tried some of the lower parameter versions of bartowski's Deepseek models (7B and 14B) but was receiving an error when trying to load the model into gpt4all. However, I tried loading in the larger model DeepSeek-R1-Distill-Llama-70B-Q5_K_S which did load in correctly and is usable. That isn't much solace to many of you who don't have the RAM but perhaps this could rule out gpt4all, possibly.

Hope this helps.

breathless19 · 2025-01-25T19:47:49Z

Same here on Bazzite Linux

kbradsha · 2025-01-25T19:54:52Z

I wonder if there is a compatible chat template generated for the 70B model which could be used for 14B or 32B even.
32B seems to be the limit taking just 21 of the 24 gigabyte available to Nvidia 90 series cards.

kbradsha · 2025-01-25T20:07:58Z

I wonder if the Llama 8B distill works as well and it's only the QWEN models experiencing the issue.

2602lim · 2025-01-27T00:28:31Z

I have the same problem. The software is unable to directly open the downloaded gguf file.
. Version 3.7.0，

fieldequation · 2025-01-27T08:39:47Z

Same here, v3.7.0, W11H. Can't load 14B and 7B model:

Could not load model due to invalid model file for DeepSeek-R1-Distill-Qwen-14B-Q8_0.gguf

fieldequation · 2025-01-27T12:45:47Z

I wonder if the Llama 8B distill works as well and it's only the QWEN models experiencing the issue.

I think so, both Llama models (8B and 70B) work fine on my device. Qwen based DeepSeek models may not compatible with current version of GPT4All

ilgrank · 2025-01-27T22:14:06Z

Also possible solution here:
https://huggingface.co/IntelligentEstate/Die_Walkure-R1-Distill-Llama-8B-iQ4_K_M-GGUF

eliuha · 2025-01-28T01:24:31Z

the solution did not work for me
version 3.7.0

realasking · 2025-01-28T02:19:01Z

Same problem here. v3.7.0, Win11. Can't load 7B, 8B, 14B DeepSeek-R1 Llama and Qwen models.

kaisernova · 2025-01-28T03:24:01Z

Same problem, Win 11

tomkpt · 2025-01-28T08:37:48Z

PLEASE MAKE DEEPSEEK compatible to GPT4All, As I had tried all GGFUF of Deepseek, but none are working.

LMStudio is working super fast and completely fine with Deepseek, so make it compatible too.

ilgrank · 2025-01-29T01:16:50Z

the solution did not work for me version 3.7.0

Edit 1: Also the 70B parameter model works, but I got 0.019T/s, which is unusable for me (too big to fit in VRAM)
Edit 2: Just the LLAMA models do work, QWEN do not.

brynrmrz · 2025-01-29T21:21:54Z

This is not an issue, GPT4ALL docs clearly say custom models are not supported OOTB and it is up to user to additional configuration to work. This issue should close.

nobody5050 · 2025-01-29T21:42:32Z

A pr was merged into the repo a few hours ago that fixes this, but if you want to get it working right now use this chat template which is based on that pr:

{%- if not add_generation_prompt is defined %}
    {%- set add_generation_prompt = false %}
{%- endif %}
{%- if messages[0]['role'] == 'system' %}
    {{- messages[0]['content'] }}
{%- endif %}
{%- for message in messages %}
    {%- if message['role'] == 'user' %}
        {{- '<｜User｜>' + message['content'] }}
    {%- endif %}
    {%- if message['role'] == 'assistant' %}
        {%- set content = message['content'].split('</think>', 1) | last %}
        {{- '<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>' }}
    {%- endif %}
{%- endfor -%}
{%- if add_generation_prompt %}
    {{- '<｜Assistant｜>' }}
{%- endif %}

tomkpt · 2025-01-30T16:55:23Z

{%- if not add_generation_prompt is defined %}
{%- set add_generation_prompt = false %}
{%- endif %}
{%- if messages[0]['role'] == 'system' %}
{{- messages[0]['content'] }}
{%- endif %}
{%- for message in messages %}
{%- if message['role'] == 'user' %}
{{- '<｜User｜>' + message['content'] }}
{%- endif %}
{%- if message['role'] == 'assistant' %}
{%- set content = message['content'].split('', 1) | last %}
{{- '<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>' }}
{%- endif %}
{%- endfor -%}
{%- if add_generation_prompt %}
{{- '<｜Assistant｜>' }}
{%- endif %}

Thanks you so much for this, it works now. But can it be possible or any plan of UI changes for Thinking Part, as it looks very good in DeepSeek online or LM Studio, the THINKING-REASONING PART should look beautifully in small box with small font with brain ICON blinking.

dasdebanna · 2025-01-30T21:18:28Z

{%- if not add_generation_prompt is defined %}
{%- set add_generation_prompt = false %}
{%- endif %}
{%- if messages[0]['role'] == 'system' %}
{{- messages[0]['content'] }}
{%- endif %}
{%- for message in messages %}
{%- if message['role'] == 'user' %}
{{- '<｜User｜>' + message['content'] }}
{%- endif %}
{%- if message['role'] == 'assistant' %}
{%- set content = message['content'].split('', 1) | last %}
{{- '<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>' }}
{%- endif %}
{%- endfor -%}
{%- if add_generation_prompt %}
{{- '<｜Assistant｜>' }}
{%- endif %}

Thanks you so much for this, it works now. But can it be possible or any plan of UI changes for Thinking Part, as it looks very good in DeepSeek online or LM Studio, the THINKING-REASONING PART should look beautifully in small box with small font with brain ICON blinking.

Are you using this model- https://huggingface.co/bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF ?

tomkpt · 2025-01-31T04:55:41Z

{%- if not add_generation_prompt is defined %}
{%- set add_generation_prompt = false %}
{%- endif %}
{%- if messages[0]['role'] == 'system' %}
{{- messages[0]['content'] }}
{%- endif %}
{%- for message in messages %}
{%- if message['role'] == 'user' %}
{{- '<｜User｜>' + message['content'] }}
{%- endif %}
{%- if message['role'] == 'assistant' %}
{%- set content = message['content'].split('', 1) | last %}
{{- '<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>' }}
{%- endif %}
{%- endfor -%}
{%- if add_generation_prompt %}
{{- '<｜Assistant｜>' }}
{%- endif %}

Thanks you so much for this, it works now. But can it be possible or any plan of UI changes for Thinking Part, as it looks very good in DeepSeek online or LM Studio, the THINKING-REASONING PART should look beautifully in small box with small font with brain ICON blinking.

Are you using this model- https://huggingface.co/bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF ?

Yes, Brother its Llama with 8B_Q4_K_M variant, its working super flawlessly.

But Just now, I came to know they release new update 3.8, so its supported natively in app now, without any custom modification, as earlier.

If you still face any issue in running it, just ping me out, I will be glad to help you in every way possible.

TS

dasdebanna · 2025-01-31T11:26:34Z

{%- if not add_generation_prompt is defined %}
{%- set add_generation_prompt = false %}
{%- endif %}
{%- if messages[0]['role'] == 'system' %}
{{- messages[0]['content'] }}
{%- endif %}
{%- for message in messages %}
{%- if message['role'] == 'user' %}
{{- '<｜User｜>' + message['content'] }}
{%- endif %}
{%- if message['role'] == 'assistant' %}
{%- set content = message['content'].split('', 1) | last %}
{{- '<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>' }}
{%- endif %}
{%- endfor -%}
{%- if add_generation_prompt %}
{{- '<｜Assistant｜>' }}
{%- endif %}

Thanks you so much for this, it works now. But can it be possible or any plan of UI changes for Thinking Part, as it looks very good in DeepSeek online or LM Studio, the THINKING-REASONING PART should look beautifully in small box with small font with brain ICON blinking.

Are you using this model- https://huggingface.co/bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF ?

Yes, Brother its Llama with 8B_Q4_K_M variant, its working super flawlessly.

But Just now, I came to know they release new update 3.8, so its supported natively in app now, without any custom modification, as earlier.

If you still face any issue in running it, just ping me out, I will be glad to help you in every way possible.

TS

Thanks you so much for this, it works now.

dantrez · 2025-01-31T13:02:26Z

Thanks, Debanna, I will give it a go.

…

On Fri, Jan 31, 2025, 6:27 AM Debanna Das ***@***.***> wrote: {%- if not add_generation_prompt is defined %} {%- set add_generation_prompt = false %} {%- endif %} {%- if messages[0]['role'] == 'system' %} {{- messages[0]['content'] }} {%- endif %} {%- for message in messages %} {%- if message['role'] == 'user' %} {{- '<｜User｜>' + message['content'] }} {%- endif %} {%- if message['role'] == 'assistant' %} {%- set content = message['content'].split('', 1) | last %} {{- '<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>' }} {%- endif %} {%- endfor -%} {%- if add_generation_prompt %} {{- '<｜Assistant｜>' }} {%- endif %} *Thanks you so much for this, it works now. But can it be possible or any plan of UI changes for Thinking Part, as it looks very good in DeepSeek online or LM Studio, the THINKING-REASONING PART should look beautifully in small box with small font with brain ICON blinking.* Are you using this model- https://huggingface.co/bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF ? Yes, Brother its Llama with *8B_Q4_K_M* variant, its working super flawlessly. But Just now, I came to know they *release new update 3.8,* so its supported natively in app now, without any custom modification, as earlier. If you still face any issue in running it, just ping me out, I will be glad to help you in every way possible. TS Thanks you so much for this, it works now. — Reply to this email directly, view it on GitHub <#3404 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXDG6JMBM5Q4U2KRVB7OJDT2NNMYHAVCNFSM6AAAAABVTZQXUOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMRWHE4DAOBXG4> . You are receiving this because you commented.Message ID: ***@***.***>

QohoZ · 2025-01-31T15:08:19Z

I encounter the error 'item at index 3 is not a prompt' with all models in the reasoning tab. If I ask just one question, it works, but if I ask another one, I get that error. I should mention that I was using it with LocalDocs; apparently, that’s when it refuses to respond until the model is reloaded.

ilgrank · 2025-01-31T17:28:35Z

@QohoZ : me too, but I think it's better if you open a separate issue. The problem you're describing is not related to this one

kbradsha added bug-unconfirmed chat gpt4all-chat issues labels Jan 22, 2025

QohoZ mentioned this issue Jan 31, 2025

LocalDocs error 'item at index 3 is not a prompt' with all models in the reasoning tab. #3445

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DeepSeek-R1 GGUF can't be loaded #3404

DeepSeek-R1 GGUF can't be loaded #3404

kbradsha commented Jan 22, 2025

dantrez commented Jan 23, 2025 •

edited

Loading

QohoZ commented Jan 24, 2025

kbradsha commented Jan 25, 2025

adgu commented Jan 25, 2025

BrushAway commented Jan 25, 2025

breathless19 commented Jan 25, 2025

kbradsha commented Jan 25, 2025

kbradsha commented Jan 25, 2025

2602lim commented Jan 27, 2025

fieldequation commented Jan 27, 2025

fieldequation commented Jan 27, 2025

ilgrank commented Jan 27, 2025

eliuha commented Jan 28, 2025

realasking commented Jan 28, 2025

kaisernova commented Jan 28, 2025

tomkpt commented Jan 28, 2025

ilgrank commented Jan 29, 2025 •

edited

Loading

brynrmrz commented Jan 29, 2025

nobody5050 commented Jan 29, 2025

tomkpt commented Jan 30, 2025 •

edited

Loading

dasdebanna commented Jan 30, 2025

tomkpt commented Jan 31, 2025

dasdebanna commented Jan 31, 2025

dantrez commented Jan 31, 2025 via email

QohoZ commented Jan 31, 2025

ilgrank commented Jan 31, 2025 •

edited

Loading

DeepSeek-R1 GGUF can't be loaded #3404

DeepSeek-R1 GGUF can't be loaded #3404

Comments

kbradsha commented Jan 22, 2025

Bug Report

Steps to Reproduce

Expected Behavior

Your Environment

dantrez commented Jan 23, 2025 • edited Loading

QohoZ commented Jan 24, 2025

kbradsha commented Jan 25, 2025

adgu commented Jan 25, 2025

BrushAway commented Jan 25, 2025

breathless19 commented Jan 25, 2025

kbradsha commented Jan 25, 2025

kbradsha commented Jan 25, 2025

2602lim commented Jan 27, 2025

fieldequation commented Jan 27, 2025

fieldequation commented Jan 27, 2025

ilgrank commented Jan 27, 2025

eliuha commented Jan 28, 2025

realasking commented Jan 28, 2025

kaisernova commented Jan 28, 2025

tomkpt commented Jan 28, 2025

ilgrank commented Jan 29, 2025 • edited Loading

brynrmrz commented Jan 29, 2025

nobody5050 commented Jan 29, 2025

tomkpt commented Jan 30, 2025 • edited Loading

dasdebanna commented Jan 30, 2025

tomkpt commented Jan 31, 2025

dasdebanna commented Jan 31, 2025

dantrez commented Jan 31, 2025 via email

QohoZ commented Jan 31, 2025

ilgrank commented Jan 31, 2025 • edited Loading

dantrez commented Jan 23, 2025 •

edited

Loading

ilgrank commented Jan 29, 2025 •

edited

Loading

tomkpt commented Jan 30, 2025 •

edited

Loading

ilgrank commented Jan 31, 2025 •

edited

Loading