Skip to content

Commit

Permalink
Gorilla OpenFunctions run locally in Google Colab (ShishirPatil#166)
Browse files Browse the repository at this point in the history
Added a notebook (titled "gorilla_openfunctions_local.ipynb") which
quantizes the Gorilla OpenFunctions model
(gorilla-llm/gorilla-openfunctions-v1) so that it can run inference
fully self-contained on Colab's free GPU. Although further inference
testing is required, the quantized local model returns the same
responses as the original hosted model for the 3 examples provided.

---------

Co-authored-by: Shishir Patil <[email protected]>
  • Loading branch information
meenakshi-mittal and ShishirPatil authored Feb 6, 2024
1 parent 092a7e0 commit c2ab5e8
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion openfunctions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,8 @@ uber.ride(loc="berkeley", type="plus", time=10)

## Running OpenFunctions Locally

You can try this out on our [Local OpenFunctions Colab](https://colab.research.google.com/drive/1I9UJoKh9sngE2MfPfQD5kbn2-twq2xvY?usp=sharing) to see how it works!

If you want to Run OpenFunctions locally, here is the prompt format that we used:

```python
Expand Down Expand Up @@ -310,7 +312,7 @@ print(resp)
```

### Try Out Self-Hosting OpenFunctions on Colab
You can try out setting up a local server to self-host the OpenFunctions model using this [Colab notebook](https://colab.research.google.com/drive/1rKbWmz6IpzHenn2-LmEnt0Zy3Mu2xwbw?usp=sharing).
You can try out setting up a local server to self-host the OpenFunctions model using this [OpenFunctions Self-Hosted Colab notebook](https://colab.research.google.com/drive/1aBxYJ9VncxDRN1-DyMT3J-ozmCRRvSje?usp=sharing).
Make sure you select an A100/V100 instance to run the notebook – smaller instances like T-4 do not suffice due to memory constraints.
To remotely access the server running on the Colab instance from a local client, ngrok is used to tunnel the server ports from the Colab instance to public URLs. Instructions for setting up ngrok are provided in the notebook.

Expand Down

0 comments on commit c2ab5e8

Please sign in to comment.