Single prompt & env overrides #23

curvedinf · 2025-01-29T01:18:48Z

No description provided.

…s not called without 'start' command

Implement --single_prompt mode to use dir-assistant as part of the workflow

curvedinf · 2025-01-29T04:25:18Z

@iSevenDays Could you test this version in your workflow?

curvedinf · 2025-01-30T17:33:17Z

@iSevenDays did you get a chance to test this version?

…tly from the repository with the currently active python bin. Run it with several argument combinations and different modes. Assume the default local LLM and embed models are installed. Assume the OpenAI API key is configured and use the 'gpt-4o-mini' and 'text-embedding-3-small' models for API tests.

…bmitted in interactive mode. Currently it assumes \n will submit, but instead it must use ALT-Enter.

…d to the test output

…dir-assistant in a virtual terminal with the current system python. Alt-Enter is used to submit prompts (as newlines are allowed). Comment the test thoroughly.

…he test output

iSevenDays · 2025-01-30T19:50:20Z

@curvedinf I can only test probably on weekend and I'm not really sure, because I'm relocating at the moment, so I can't promise.
I did test my version, the only thing that needs additional testing is whether GLOBAL_IGNORE parameter works and dir-assistant config, I didn't test that.
--no-color works properly and no longer produces strange symbols when using from API.
Your version looks much cleaner, thanks!

Overriding Configurations with Environment Variables

I see however some missing items like
GLOBAL_IGNORE should be available - some projects have big .js files like 10mb of some static jQuery library where dependencies put together and don't provide useful context with openai/bge-m3 takes indefinite amount of time to generate embeddings for not useful data.
LITELLM_CONTEXT size is a must, because I test locally and with runpod.
For local testing I set context size of 4-12k depending on the model name e.g. dynamically.
With runpod I set around 128k.

LiteLLM OpenAI API KEY & Model must be also overridable, I'm not sure this version supports this.
litellm_embed_model is also set dynamically by my server.py (that calls dir-assistant internally).

I hope you find my feedback useful.

curvedinf · 2025-01-30T19:57:51Z

No sweat, thanks for the update. Let me know if you get around to it. I'd like to release this next week.

curvedinf · 2025-01-30T20:01:54Z

Good point on GLOBAL_IGNORES. I'll see if I can get that in.

curvedinf · 2025-01-30T20:07:58Z

All other config settings should be working in this branch. They are under a new scheme from your branch though. They now live in a namespace so they won't collide with other env vars. Also all config subgroups live in their own sub-namespace. Some examples:

export DIR_ASSISTANT__LITELLM_MODEL="gpt-4o-mini"
export DIR_ASSISTANT__LITELLM_CONTEXT=20000
export DIR_ASSISTANT__LITELLM_API_KEYS__OPENAI_API_KEY="abcd1234"

Note on API Keys: LiteLLM by default uses env variables to collect the API keys, and that's the mechanism dir-assistant uses to set them for LiteLLM. Dir-assistant checks if any of the API keys in the config are already set in the environment and does not overwrite them if they exist, so you can always set just OPENAI_API_KEY instead of the dir-assistant config method.

iSevenDays · 2025-02-02T13:59:38Z

I have just tested the project and it seems I can't override OPENAI_API_KEY and OPENAI_API_BASE

Traceback (most recent call last):
  File "/usr/local/bin/dir-assistant", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/dir_assistant/main.py", line 224, in main
    run_single_prompt(args, config_dict["DIR_ASSISTANT"])
  File "/usr/local/lib/python3.12/site-packages/dir_assistant/cli/start.py", line 47, in run_single_prompt
    llm = initialize_llm(args, config_dict)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/dir_assistant/cli/start.py", line 202, in initialize_llm
    index, chunks = create_file_index(embed, ignore_paths, embed_chunk_size, extra_dirs, args.verbose)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/dir_assistant/assistant/index.py", line 108, in create_file_index
    file_chunks, file_embeddings = process_file(
                                   ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/dir_assistant/assistant/index.py", line 181, in process_file
    embedding = embed.create_embedding(chunk_header + current_chunk)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/dir_assistant/assistant/lite_llm_embed.py", line 17, in create_embedding
    return embedding(model=self.lite_llm_model, input=text, timeout=600)["data"][0][
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/litellm/utils.py", line 1080, in wrapper
    raise e
  File "/usr/local/lib/python3.12/site-packages/litellm/utils.py", line 958, in wrapper
    result = original_function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/litellm/main.py", line 3772, in embedding
    raise exception_type(
          ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2190, in exception_type
    raise e
  File "/usr/local/lib/python3.12/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 1120, in exception_type
    raise AuthenticationError(
litellm.exceptions.AuthenticationError: litellm.AuthenticationError: geminiException - {
  "error": {
    "code": 400,
    "message": "API key not valid. Please pass a valid API key.",
    "status": "INVALID_ARGUMENT",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "API_KEY_INVALID",
        "domain": "googleapis.com",
        "metadata": {
          "service": "generativelanguage.googleapis.com"
        }
      },
      {
        "@type": "type.googleapis.com/google.rpc.LocalizedMessage",
        "locale": "en-US",
        "message": "API key not valid. Please pass a valid API key."
      }
    ]
  }
}

appuser@dir-assistant-server-787c76d88c-ndbt7:/workspace/projects/empty$ OPENAI_API_KEY='empty' OPENAI_API_BASE="https://xxxx1368.ngrok-free.app/v1" dir-assistant --single-prompt "what is this project about?"

I have tried
DIR_ASSISTANT__LITELLM_EMBED_MODEL="openai/bge-m3" OPENAI_API_KEY='empty' OPENAI_API_BASE="https://0xxx68.ngrok-free.app/v1" dir-assistant --single-prompt "what is this project about?"

and I tried
DIR_ASSISTANT__LITELLM_API_KEYS__OPENAI_API_KEY="empty" DIR_ASSISTANT__LITELLM_EMBED_MODEL="openai/bge-m3" OPENAI_API_KEY='empty' OPENAI_API_BASE="https://xxxx.ngrok-free.app/v1" dir-assistant --single-prompt "what is this project about?"

curvedinf · 2025-02-02T15:57:29Z

Can you try with another litellm version? The error says it is using their gemini adapter and gemini key, not openai. Dir-assistant just passes the model string in and litellm picks the api and key to use.

curvedinf · 2025-02-02T16:26:56Z

Also, is your config file api set to gemini? It could be a bug where it isn't applying the env vars

iSevenDays · 2025-02-02T19:53:35Z

@curvedinf no, I don't use gemini. I only use OpenAI-Compatible server (e.g. Ollama or LM Studio).
I posted my env vars. Even when I enter the pod(or vm), and execute dir-assistant from this branch directly with OPENAI key and base url, dir-assistant tries to use llm with gemini.

here is a verbose log

appuser@dir-assistant-server-787c76d88c-ndbt7:/workspace/projects/empty$ DIR_ASSISTANT__LITELLM_API_KEYS__OPENAI_API_KEY="empty" DIR_ASSISTANT__LITELLM_EMBED_MODEL="openai/bge-m3" OPENAI_API_KEY='empy' OPENAI_API_BASE="https://02d4-2a02-810a-a82-a400-2074-61a8-59b1-1368.ngrok-free.app/v1" dir-assistant --single-prompt "what is this project about?" --verbose
/usr/local/lib/python3.12/site-packages/pydantic/_internal/_config.py:334: UserWarning: Valid config keys have changed in V2:
* 'fields' has been removed
  warnings.warn(message, UserWarning)
Debug: Applying config override LITELLM_API_KEYS={'OPENAI_API_KEY': 'empy', 'DIR_ASSISTANT__LITELLM_API_KEYS__OPENAI_API_KEY': 'empty'}
Loading embedding model...
Creating file embeddings and index...
Finding files to index...
Creating embeddings for /workspace/projects/empty/first-file.txt

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/litellm/main.py", line 3505, in embedding
    response = google_batch_embeddings.batch_embeddings(  # type: ignore
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/litellm/llms/vertex_ai/gemini_embeddings/batch_embed_content_handler.py", line 118, in batch_embeddings
    response = sync_handler.post(
               ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/litellm/llms/custom_httpx/http_handler.py", line 535, in post
    raise e
  File "/usr/local/lib/python3.12/site-packages/litellm/llms/custom_httpx/http_handler.py", line 516, in post
    response.raise_for_status()
  File "/usr/local/lib/python3.12/site-packages/httpx/_models.py", line 763, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://generativelanguage.googleapis.com/v1beta/models/text-embedding-004:batchEmbedContents?key=None'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/dir-assistant", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/dir_assistant/main.py", line 224, in main
    run_single_prompt(args, config_dict["DIR_ASSISTANT"])
  File "/usr/local/lib/python3.12/site-packages/dir_assistant/cli/start.py", line 47, in run_single_prompt
    llm = initialize_llm(args, config_dict)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/dir_assistant/cli/start.py", line 202, in initialize_llm
    index, chunks = create_file_index(embed, ignore_paths, embed_chunk_size, extra_dirs, args.verbose)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/dir_assistant/assistant/index.py", line 108, in create_file_index
    file_chunks, file_embeddings = process_file(
                                   ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/dir_assistant/assistant/index.py", line 181, in process_file
    embedding = embed.create_embedding(chunk_header + current_chunk)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/dir_assistant/assistant/lite_llm_embed.py", line 17, in create_embedding
    return embedding(model=self.lite_llm_model, input=text, timeout=600)["data"][0][
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/litellm/utils.py", line 1080, in wrapper
    raise e
  File "/usr/local/lib/python3.12/site-packages/litellm/utils.py", line 958, in wrapper
    result = original_function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/litellm/main.py", line 3772, in embedding
    raise exception_type(
          ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2190, in exception_type
    raise e
  File "/usr/local/lib/python3.12/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 1120, in exception_type
    raise AuthenticationError(
litellm.exceptions.AuthenticationError: litellm.AuthenticationError: geminiException - {
  "error": {
    "code": 400,
    "message": "API key not valid. Please pass a valid API key.",
    "status": "INVALID_ARGUMENT",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "API_KEY_INVALID",
        "domain": "googleapis.com",
        "metadata": {
          "service": "generativelanguage.googleapis.com"
        }
      },
      {
        "@type": "type.googleapis.com/google.rpc.LocalizedMessage",
        "locale": "en-US",
        "message": "API key not valid. Please pass a valid API key."
      }
    ]
  }
}

appuser@dir-assistant-server-787c76d88c-ndbt7:/workspace/projects/empty$

iSevenDays · 2025-02-02T21:22:08Z

I found a config that works. The documentation or the code contains bug.
Here is working version:

LITELLM_MODEL="openai/glm-4-9b-chat-abliterated" LITELLM_EMBED_MODEL="openai/bge-m3" A
CTIVE_EMBED_IS_LOCAL=false ACTIVE_MODEL_IS_LOCAL=false DIR_ASSISTANT__LITELLM_API_KEYS__OPENAI_API_KEY="empty" DIR_ASSISTANT__LITELLM_EMBED_MODEL="openai/bge-m
3" OPENAI_API_KEY='empy' OPENAI_API_BASE="https://xxxx.ngrok-free.app/v1" dir-assistant --single-prompt "what is this project about?" --verbose 
/usr/local/lib/python3.12/site-packages/pydantic/_internal/_config.py:334: UserWarning: Valid config keys have changed in V2:
* 'fields' has been removed
  warnings.warn(message, UserWarning)
Debug: Applying config override ACTIVE_MODEL_IS_LOCAL=False
Debug: Applying config override ACTIVE_EMBED_IS_LOCAL=False
Debug: Applying config override LITELLM_MODEL=openai/glm-4-9b-chat-abliterated
Debug: Applying config override LITELLM_API_KEYS={'OPENAI_API_KEY': 'empy', 'DIR_ASSISTANT__LITELLM_API_KEYS__OPENAI_API_KEY': 'empty'}
Debug: Applying config override LITELLM_EMBED_MODEL=openai/bge-m3
Loading embedding model...
Creating file embeddings and index...
Finding files to index...
Using cached embeddings for /workspace/projects/empty/first-file.txt
Creating index from embeddings...

iSevenDays · 2025-02-02T21:25:27Z

LITELLM_CONTEXT_SIZE=9999 is also a only way to override litellm context size, not DIR_ASSISTANT__LITELLM_CONTEXT_SIZE=9999

Checks if the filepath either: Ends with the ignore pattern (e.g., resources/swagger/) Contains the ignore pattern as a path component

iSevenDays and others added 26 commits January 16, 2025 18:37

feat: Add single prompt mode to dir-assistant

214c7b1

feat: Add variables for file watcher and startup art initialization

5b2f8fc

feat: format single-prompt output with colors and spacing

1160ed0

Implement single prompt mode for agentic usage

c23230d

Leave gitignore as is

f8cd9d4

Allow --single_prompt to be called from the top level and fix if it i…

2ea7adb

…s not called without 'start' command

Update config parsing

1a1056d

Update config parsing

47ea140

Fix verbose logging

5688809

Fix setting of OPENAI_API_KEY in env

4820c21

Correctly override LITELLM keys

ddaba42

Add --no-color option for agentic usage

2607157

Add proper --no-color handling across codebase for agentic usage

5cacbaa

Improve no-color handling

60f7121

Add --no_color to 'start' command

25f59c2

Merge pull request #20 from iSevenDays/feature/single_prompt_mode_imp

188e560

Implement --single_prompt mode to use dir-assistant as part of the workflow

progress cleaning up changes

6b31805

readme edits

3eb69bd

progress

b4d607f

updated new features

9c1b3ba

Indexing files message while verbose is off

2bad8d7

annoying pydantic warning from a dependency removed

9c6f67f

Working well?

5843f2d

formatted and readme additions

14db98d

readme fixes

8d1c467

removed --config

574cf9e

downloads

cecd041

curvedinf requested a review from iSevenDays January 30, 2025 17:33

curvedinf added 8 commits January 30, 2025 12:16

modify test/test_smoketest.py so it uses pytest instead of unittest.

01579d4

Update test_smoketest.py with the changes you specified

c342882

update test_smoketest.py with pty

05de0bc

modify test_smoketest.py so it fixes an issue with how a prompt is su…

bd5524b

…bmitted in interactive mode. Currently it assumes \n will submit, but instead it must use ALT-Enter.

modify test_smoketest.py so output from the virtual terminal is route…

d4a712b

…d to the test output

write the test for test/test_smoketest.py. Use pytest and pty to run …

f0caf84

…dir-assistant in a virtual terminal with the current system python. Alt-Enter is used to submit prompts (as newlines are allowed). Comment the test thoroughly.

Modify test_smoketest.py so it prints all pseudo-terminal output to t…

2ebf122

…he test output

curvedinf added 2 commits January 30, 2025 14:26

added smoketests and fixed a bug with env var override type coercion

53d3c89

formatted code

cb968b8

iSevenDays and others added 4 commits February 12, 2025 16:22

Fixed --ignore

7d786e4

Checks if the filepath either: Ends with the ignore pattern (e.g., resources/swagger/) Contains the ignore pattern as a path component

Implement proper path ignoring

db534d6

Reverting ignore method

6ab1d7c

fixed anthropic models

9a7a048

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Single prompt & env overrides #23

Single prompt & env overrides #23

curvedinf commented Jan 29, 2025

curvedinf commented Jan 29, 2025

curvedinf commented Jan 30, 2025

iSevenDays commented Jan 30, 2025 •

edited

Loading

curvedinf commented Jan 30, 2025

curvedinf commented Jan 30, 2025

curvedinf commented Jan 30, 2025 •

edited

Loading

iSevenDays commented Feb 2, 2025

curvedinf commented Feb 2, 2025

curvedinf commented Feb 2, 2025

iSevenDays commented Feb 2, 2025

iSevenDays commented Feb 2, 2025

iSevenDays commented Feb 2, 2025

Single prompt & env overrides #23

Are you sure you want to change the base?

Single prompt & env overrides #23

Conversation

curvedinf commented Jan 29, 2025

curvedinf commented Jan 29, 2025

curvedinf commented Jan 30, 2025

iSevenDays commented Jan 30, 2025 • edited Loading

Overriding Configurations with Environment Variables

curvedinf commented Jan 30, 2025

curvedinf commented Jan 30, 2025

curvedinf commented Jan 30, 2025 • edited Loading

iSevenDays commented Feb 2, 2025

curvedinf commented Feb 2, 2025

curvedinf commented Feb 2, 2025

iSevenDays commented Feb 2, 2025

iSevenDays commented Feb 2, 2025

iSevenDays commented Feb 2, 2025

iSevenDays commented Jan 30, 2025 •

edited

Loading

curvedinf commented Jan 30, 2025 •

edited

Loading