[Feature request]: support for azure hosted openai models?

[mariusgarmhausen]

What do you need?

I was wondering if Azure hosted endpoints are already implemented?

[streichsbaer]

I was also looking for the Azure OpenAI option, but it's not yet supported.

@danielmiessler, I'd be happy to create a PR for that feature.

Let me know.

[GuizzyQC]

Try adding a OPENAI_BASE_URL= line in ~/.config/fabric/.env

It does complain about "Unable to fetch models" and "No OpenAI API key found" (despite having an API key too), as elsewhere in the code it does fetch a list of models from a hardcoded OpenAI URL, but outside of that it does work for me on a different OpenAI API compatible endpoint.

[bjcbusiness]

Try adding a OPENAI_BASE_URL= line in ~/.config/fabric/.env

It does complain about "Unable to fetch models" and "No OpenAI API key found" (despite having an API key too), as elsewhere in the code it does fetch a list of models from a hardcoded OpenAI URL, but outside of that it does work for me on a different OpenAI API compatible endpoint.

This doesn't work at all for me. Looking forward to proper support for Azure.

[vfarcic]

👍 for Azure OpenAI support

[AtheeSouhait]

Made it working with my Azure openai deployment, you can test:
https://github.com/AtheeSouhait/fabric/tree/AtheeSouhait-AzureOpenAIonCLI

New environment variables must contain:
AZURE_OPENAI_API_KEY
AZURE_OPENAI_ENDPOINT
AZURE_OPENAI_API_VERSION
AZURE_OPENAI_MODEL
for example:
export AZURE_OPENAI_API_VERSION=2024-02-15-preview
export AZURE_OPENAI_ENDPOINT=https://yourdeploymentendpoint.openai.azure.com/
export AZURE_OPENAI_MODEL=myGPT4-32k
export AZURE_OPENAI_API_KEY=yourazureopenaideploymentapikey

[bjcbusiness]

Works for me. I set the required environment variables in the .env file. Able to list models and use a pattern. Thank you very much. From: AtheeSouhait ***@***.***> Sent: Saturday, June 1, 2024 5:39 PM To: danielmiessler/fabric ***@***.***> Cc: Collar, Ben (DI SW PLM LCS CF) ***@***.***>; Comment ***@***.***> Subject: Re: [danielmiessler/fabric] [Feature request]: support for azure hosted openai models? (Issue #179) Made it working with my Azure openai deployment, you can test: https://github.com/AtheeSouhait/fabric/tree/AtheeSouhait-AzureOpenAIonCLI New environment variables must contain: AZURE_OPENAI_API_KEY AZURE_OPENAI_ENDPOINT AZURE_OPENAI_API_VERSION AZURE_OPENAI_MODEL for example: export AZURE_OPENAI_API_VERSION=2024-02-15-preview export AZURE_OPENAI_ENDPOINT=https://yourdeploymentendpoint.openai.azure.com/ export AZURE_OPENAI_MODEL=myGPT4-32k export AZURE_OPENAI_API_KEY=yourazureopenaideploymentapikey - Reply to this email directly, view it on GitHub<#179 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BHHDEZ7ECXS5RVD2DB5BUCTZFH2MDAVCNFSM6AAAAABERBCBASVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBTGUYDQNZSHA>. You are receiving this because you commented.Message ID: ***@***.******@***.***>>

[stefanstranger]

Hi @AtheeSouhait, I followed your instructions and copied your code changes but I don't see the model being showed when running fabric --listmodels.

What else do I need to change to support Azure OpenAI?

Thanks for your help.
/Stefan

[AtheeSouhait]

Hi @stefanstranger , indeed the update don't modify output of --listmodel so it is not included in the list. But your setting automaticaly uses your AZURE open ai model. By the way, if you want to use OPENAI model and not the Azure; comment the AZURE_OPENAI_API_KEY line of your .env so it is not taken as default.
I should have also adapted this "listmodel" part of the code. I'll do if I have the time but as the project will move to "go" language implementation, I'm not sure it worth the effort, what do you think?

[stefanstranger]

Hi @AtheeSouhait,

Thanks for your quick response 🙏.

I've commented out the OPENAI_API_KEY for now to be sure it will use the AZURE_OPENAI_API_KEY.

When I now try use Fabric I get the following error.

echo "What is the Capital of the Netherlands?" | fabric --stream --pattern ai
Error: 'NoneType' object has no attribute 'chat'
'NoneType' object has no attribute 'chat'

When running the following curl I just get a response:

curl "https://<instancename>.openai.azure.com/openai/deployments/chatdemo/chat/completions?api-version=2024-02-15-preview" \
  -H "Content-Type: application/json" \
  -H "api-key: <enter api key>" \
  -d "{
  \"messages\": [{\"role\":\"system\",\"content\":\"What is the capital of the Netherlands?\"}],
  \"max_tokens\": 800,
  \"temperature\": 0.7,
  \"frequency_penalty\": 0,
  \"presence_penalty\": 0,
  \"top_p\": 0.95,
  \"stop\": null
}"

Something still is not configured correctly.

What am I missing? Is there a way to list this model?

Added note:

• Running this on WSL Ubuntu.

[AtheeSouhait]

@stefanstranger did you get my repo's code? because I can"t see my changes any more in the official version. Or you can see what I changed in the utils.py file and report in the repo you cloned.

[stefanstranger]

Yes I copied the raw content from 'your' utils.py file to my local repo.

[AtheeSouhait]

Not working when I use --stream but working without.

Is it working when you use openai and not azure Open ai (comment the AZURE_OPENAI_API_KEY of env) ?

[stefanstranger]

Tried without using --stream switch but that is also giving me the same error.

Error: 'NoneType' object has no attribute 'chat'
'NoneType' object has no attribute 'chat'

It is working when I use OpenAI but not when I use Azure OpenAI.

Very strange that I cannot get this working with Azure OpenAI.

[MrPewPewLaser]

@stefanstranger I got it to work for me. The problem seems to be that his Code is far behind the original repo that requires newer package versions. So I cloned his entire repo and forced a new install of fabric using pipx install . --force

Hope it helps. :)

[stefanstranger]

Hi @MrPewPewLaser thanks for your help.

Will try your suggestions and let you know if that works.

Regards,
Stefan

[AtheeSouhait]

Thanks @MrPewPewLaser, @stefanstranger : I also re-tested : resinstallation of fabric from https://github.com/danielmiessler/fabric (cloned, put my utils.py, and continue all installation steps + add Azure environment variables). And after the installation it worked (with gpt4 model in Azure , I didn't try 3.5 or other models yet). I know it is not perfect as --stream seams not to be supported. I hope it will work for you.

[stefanstranger]

I got it to work. One more lesson learned during the setup of the Azure OpenAI configuration is that is need the Azure OpenAI deployment name and NOT the Model name in the configuration.

[AtheeSouhait]

@stefanstranger @MrPewPewLaser Great, I'm working on a dockerfile to ease the installation, setup, and the usage. I let you know.

[eugeis]

Azure Open AI provider is supported now.

[josearias210]

@stefanstranger @MrPewPewLaser @AtheeSouhait I'm trying to implement Azure OpenIA as a vendor, and I'm using the docker version.

Is there any progress on this? Is there a branch where there is a development?

I entered the container and this is the error I get, although the .env file is configured correctly

[AtheeSouhait]

Hi, @josearias210
If you have the "go" version of fabric, test with those environment variable & names, that are not the same as yours:
DEFAULT_VENDOR=Azure
DEFAULT_MODEL=gpt-4o
PATTERNS_LOADER_GIT_REPO_URL=https://github.com/danielmiessler/fabric.git
PATTERNS_LOADER_GIT_REPO_PATTERNS_FOLDER=patterns
AZURE_API_KEY=xxxx
AZURE_API_BASE_URL=https://xxx.openai.azure.com/
AZURE_DEPLOYMENTS=gpt-4o
YOUTUBE_API_KEY=xxx

[josearias210]

@AtheeSouhait Thanks for your reply, I made the change and I think I made some progress because now I am facing this error.

I don't have any pattern. It seems like a bug in Dockerfile trying to copy the patterns?

[josearias210]

@AtheeSouhait @stefanstranger

I manually copied the files that are created /patterns into root/.config/fabric/patterns and this fixed that problem