LibreChat responds with nonsense when calling LiteLLM as Proxy for Ollama

[K-J-VV]

What happened?

When trying to call LiteLLM acting as a proxy for Ollama, the following is how it responds

Versus an actual response when I call Ollama directly

Steps to Reproduce

I am running LibreChat with an OpenID setup for login/security. Otherwise my setup is default instances of NGINX Proxy Manager, LibreChat, LiteLLM and Ollama.

Based on my error logs for LiteLLM I'm thinking this may be something to do with authentication/permissions? I'm not sure if this is related and am only thinking its related because the error log specifically is pointing out my reverse proxy manager internal ip. I do have all the instances at a HTTPS domain of librechat.example.com litellm.example.com etc etc

What browsers are you seeing the problem on?

No response

Relevant log output

Here is the log from LiteLLM (the IP of 10.10.10.10 is that of my internal instance of NGINX Proxy Manager)

 Thank you for using LiteLLM! - Krrish & Ishaan
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM: Proxy initialized with Config, Set models:
    gpt-3.6-turbo
    llama23
LiteLLM_VerificationTokenView Exists!
MonthlyGlobalSpend Exists!
Last30dKeysBySpend Exists!
Last30dModelsBySpend Exists!
MonthlyGlobalSpendPerKey Exists!
Last30dTopEndUsersSpend Exists!
INFO:     10.10.10.10:33202 - "GET /v1/models HTTP/1.1" 401 Unauthorized
INFO:     10.10.10.10:45250 - "GET /v1//models HTTP/1.1" 404 Not Found
INFO:     10.10.10.10:45252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.10.10.10:34430 - "GET /v1//models HTTP/1.1" 404 Not Found
INFO:     10.10.10.10:34432 - "POST /v1/chat/completions HTTP/1.1" 200 OK


Here is the error from LibreChat

2024-03-26 12:46:06 error: Failed to fetch models from LiteLLM API The request was made and the server responded with a status code that falls out of the range of 2xx: Request failed with status code 404
2024-03-26 12:46:06 error: [MeiliMongooseModel.findOneAndUpdate] Convo not found in MeiliSearch and will index 8fe3852d-cb06-43ca-aeb2-ef7ef550ac60 Document `8fe3852d-cb06-43ca-aeb2-ef7ef550ac60` not found.
2024-03-26 12:46:13 warn: [OpenAIClient.chatCompletion][finalChatCompletion] Aborted Message
2024-03-26 12:46:13 warn: [OpenAIClient.chatCompletion][finalChatCompletion] API error

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[K-J-VV]

So to experiment, I changed the Ollama LLM from llama2 to dolphin-mistral. This is the only thing I changed and now here is the new response from LiteLLM

Again, the call directly to Ollama responds normally, but when through LiteLLM the response is... odd

[danny-avila]

I'm not too familiar with how LiteLLM is handling the proxy for Ollama, but they could be passing/interpreting params from LibreChat that have maybe different corresponding values as opposed to Ollama direct.

Have you experimented with adding/dropping default params?

These are the default params that LibreChat includes in the API call.

https://docs.librechat.ai/install/configuration/custom_config.html#breakdown-of-default-params

They can be modified or removed using addParams and dropParams:
https://docs.librechat.ai/install/configuration/custom_config.html#addparams

[K-J-VV]

I'm not too familiar with how LiteLLM is handling the proxy for Ollama, but they could be passing/interpreting params from LibreChat that have maybe different corresponding values as opposed to Ollama direct.

Have you experimented with adding/dropping default params?

These are the default params that LibreChat includes in the API call.

https://docs.librechat.ai/install/configuration/custom_config.html#breakdown-of-default-params

They can be modified or removed using addParams and dropParams: https://docs.librechat.ai/install/configuration/custom_config.html#addparams

It seems that LiteLLM only works when parameters are dropped. When I set no parameters i have the following error

The following is what i have for the config file to receive the garbled/non-sense

model_list:
  - model_name: gpt-3.5-turbo # user-facing model alias
    litellm_params:
      model: ollama/llama2
      api_base: https://ollama.example.com/
      stream: True

litellm_settings:
  drop_params: True

general_settings: 
  master_key: sk-xxxxx # [OPTIONAL] if set all calls to proxy will require either this key or a valid generated token

[danny-avila]

Maybe @bsu3338 can weigh in who uses litellm a lot. I have to set it up, since I don't use it, in order to test.

I suppose you could just drop presence_penalty from librechat side in order to include the rest of the parameters from librechat.

Add this to your librechat.yaml config, in the ollama section

dropParams: ["presence_penalty"]

[danny-avila]

You can also dial in the settings that ollama will accept (not sure which they are):

addParams:
      temperature: 0.5

[bsu3338]

Instead of ollama/llama2 try openai/llama2. Just curious it using ollama openai api compatibility makes a difference. May have to add /v1 to your base.

[K-J-VV]

Instead of ollama/llama2 try openai/llama2. Just curious it using ollama openai api compatibility makes a difference. May have to add /v1 to your base.

This worked! I did both suggestions: change to openai/llama2 and added /v1 to my base

Another question..

In my LibreChat config I only have 2x endpoints specified (LiteLLM and Ollama), but in my LibreChat UI OpenAi displays as an option. The LiteLLM and Ollama endpoints work, the OpenAI one does not (assuming because it is not mapped to anything.) Any suggestions on how to hide/remove the OpenAi option?

I believe it shows because LiteLLM requires the .env variable of OPENAI_API_KEY to be set (any random variable).

[danny-avila]

Glad it worked!

I believe it shows because LiteLLM requires the .env variable of OPENAI_API_KEY to be set (any random variable).

It doesn't need the OPENAI_API_KEY set. You can use any arbitrary name/value as the key when using the custom config file.

For example:

    - name: "LiteLLM"
      apiKey: "none"

Try removing the OPENAI_API_KEY altogether after that.

If anything you can use the ENDPOINTS env var

ENDPOINTS=custom

When I set it like this, only my custom endpoints show up:

[K-J-VV]

So it may actually be that LiteLLM requires the OPENAI_API_KEY variable to be set, see: https://litellm.vercel.app/docs/proxy_server#tutorial-use-with-multiple-llms--librechatchatbot-uiauto-genchatdevlangroidetc

This fix doesn't work when I do not have that variable set (LiteLLM won't start properly). The error I get in LiteLLM is

openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable

It appears both projects have identically named variables, and maybe it becomes a requirement because my litellm config only works in this setup when i change from ollama/llama2 to openai/llama2?

I like to keep everything in a .env file, so it looks like both pojects picked it up. For now as a fix, i am just passing the environment OPENAI_API_KEY specifically to the LiteLLM component of my dockerfile and all is good!

[danny-avila]

Great, yeah if the containers share .env file then I could see that. Glad you figured it out