Merge remote-tracking branch 'origin/master' into add_more_scorers

.github/workflows/pr.yaml

@@ -1,6 +1,7 @@
-name: Validate PR
+name: Check PR
 
 on:
+  workflow_dispatch:
   pull_request_target:
     types: [opened, edited, synchronize]
 
@@ -10,26 +11,27 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       # https://github.com/amannn/action-semantic-pull-request/releases
-      - uses: amannn/action-semantic-pull-request@v4.2.0
+      - uses: amannn/action-semantic-pull-request@v5
         env:
           GITHUB_TOKEN: ${{ secrets.PR_TITLE_GITHUB_TOKEN }}
         with:
-          # Allowed types: https://www.notion.so/wandbai/Pull-Requests-17966b997841407b95b7c36414ae0634
+          # Allowed types: See CONTRIBUTING.md
           types: |
+            chore
+            docs
             feat
             fix
-            docs
-            style
-            refactor
             perf
-            test
-            analytics
-            build
-            ci
-            chore
+            refactor
             revert
+            style
             security
-            release
+            test
+          scopes: |
+            ui
+            weave
+            weave_query
           wip: false
+          requireScope: true
           validateSingleCommit: false
           validateSingleCommitMatchesPrTitle: false

.github/workflows/test.yaml

@@ -231,6 +231,7 @@ jobs:
             'cohere',
             'dspy',
             'groq',
+            'google_ai_studio',
             'instructor',
             'langchain',
             'litellm',
@@ -290,6 +291,7 @@ jobs:
           WB_SERVER_HOST: http://wandbservice
           WF_CLICKHOUSE_HOST: weave_clickhouse
           WEAVE_SERVER_DISABLE_ECOSYSTEM: 1
+          GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
         run: |
           nox -e "tests-${{ matrix.python-version-major }}.${{ matrix.python-version-minor }}(shard='${{ matrix.nox-shard }}')"
   trace-tests-matrix-check: # This job does nothing and is only used for the branch protection

.gitignore

@@ -7,9 +7,6 @@ __pycache__
 .idea
 *.egg-info/
 build/
-MNIST
-food-101
-food-101.tar.gz
 gha-creds-*.json
 .vscode
 .mypy_cache

CONTRIBUTING.md

@@ -15,7 +15,7 @@
 ## Issues
 1. Check the [issues](https://github.com/wandb/weave/issues) and [PRs](https://github.com/wandb/weave/pulls) to see if the feature/bug has already been requested/fixed. If not, [open an issue](https://github.com/wandb/weave/issues/new/choose). This helps us keep track of feature requests and bugs!
 2. If you're having issues, the best way we can help is when you can share a reproducible example.
-   1. In general, it's helpful use this format:
+   1. In general, it's helpful to use this format:
       ```
       <short description of the issue>
       <link to your project>

docs/CONTRIBUTING_DOCS.md

@@ -115,7 +115,7 @@ See `docs/scripts/generate_service_api_spec.py` and `./docs/reference/service-ap
 
 Service doc generation loads the `openapi.json` file describing the server, processes it, then uses the `docusaurus-plugin-openapi-docs` plugin to generate markdown files from that specification.
 
-To improve docs, basically follow FastAPI's instructions to create good Swagger docs by adding field-level and endpoint-level documentation using their APIs. Assuming your have made some changes, `docs/scripts/generate_service_api_spec.py` needs a server to point to. You can either deploy to prod, or run the server locally and point to it in `docs/scripts/generate_service_api_spec.py`. From there, `docs/scripts/generate_service_api_spec.py` will download the spec, clean it up, and build the docs!
+To improve docs, basically follow FastAPI's instructions to create good Swagger docs by adding field-level and endpoint-level documentation using their APIs. Assuming you have made some changes, `docs/scripts/generate_service_api_spec.py` needs a server to point to. You can either deploy to prod, or run the server locally and point to it in `docs/scripts/generate_service_api_spec.py`. From there, `docs/scripts/generate_service_api_spec.py` will download the spec, clean it up, and build the docs!
 
 ### Notebook Doc Gen
 

docs/docs/guides/integrations/google-gemini.md

@@ -2,22 +2,106 @@
 
 Google offers two ways of calling Gemini via API:
 
-1. Via the Vertex APIs ([docs](https://cloud.google.com/vertexai/docs))
-2. Via the Gemini API ([docs](https://ai.google.dev/gemini-api/docs/quickstart?lang=python))
+1. Via the [Vertex APIs](https://cloud.google.com/vertexai/docs).
+2. Via the [Gemini API SDK](https://ai.google.dev/gemini-api/docs/quickstart?lang=python).
 
-## Vertex API
+## Tracing
 
-Full Weave support for the `Vertex AI SDK` python package is currently in development, however there is a way you can integrate Weave with the Vertex API. 
+It’s important to store traces of language model applications in a central location, both during development and in production. These traces can be useful for debugging, and as a dataset that will help you improve your application.
 
-Vertex API supports OpenAI SDK compatibility ([docs](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/call-gemini-using-openai-library)), and if this is a way you build your application, Weave will automatically track your LLM calls via our [OpenAI](/guides/integrations/openai) SDK integration.
+Weave will automatically capture traces for [Gemini API SDK](https://ai.google.dev/gemini-api/docs/quickstart?lang=python). To start tracking, calling `weave.init(project_name="<YOUR-WANDB-PROJECT-NAME>")` and use the library as normal.
 
-\* Please note that some features may not fully work as Vertex API doesn't implement the full OpenAI SDK capabilities.
+```python
+import os
+import google.generativeai as genai
+import weave
+
+weave.init(project_name="google_ai_studio-test")
+
+genai.configure(api_key=os.environ["GOOGLE_API_KEY"])
+model = genai.GenerativeModel("gemini-1.5-flash")
+response = model.generate_content("Write a story about an AI and magic")
+```
+
+## Track your own ops
+
+Wrapping a function with `@weave.op` starts capturing inputs, outputs and app logic so you can debug how data flows through your app. You can deeply nest ops and build a tree of functions that you want to track. This also starts automatically versioning code as you experiment to capture ad-hoc details that haven't been committed to git.
+
+Simply create a function decorated with [`@weave.op`](/guides/tracking/ops).
+
+In the example below, we have the function `recommend_places_to_visit` which is a function wrapped with `@weave.op` that recommends places to visit in a city.
+
+```python
+import os
+import google.generativeai as genai
+import weave
+
+weave.init(project_name="google_ai_studio-test")
+genai.configure(api_key=os.environ["GOOGLE_API_KEY"])
+
+
+@weave.op()
+def recommend_places_to_visit(city: str, model: str = "gemini-1.5-flash"):
+    model = genai.GenerativeModel(
+        model_name=model,
+        system_instruction="You are a helpful assistant meant to suggest all budget-friendly places to visit in a city",
+    )
+    response = model.generate_content(city)
+    return response.text
+
+
+recommend_places_to_visit("New York")
+recommend_places_to_visit("Paris")
+recommend_places_to_visit("Kolkata")
+```
+
+## Create a `Model` for easier experimentation
 
-## Gemini API
+Organizing experimentation is difficult when there are many moving pieces. By using the [`Model`](../core-types/models) class, you can capture and organize the experimental details of your app like your system prompt or the model you're using. This helps organize and compare different iterations of your app. 
 
-:::info
+In addition to versioning code and capturing inputs/outputs, [`Model`](../core-types/models)s capture structured parameters that control your application’s behavior, making it easy to find what parameters worked best. You can also use Weave Models with `serve`, and [`Evaluation`](../core-types/evaluations.md)s.
 
-Weave native client integration with the `google-generativeai` python package is currently in development
-:::
+In the example below, you can experiment with `CityVisitRecommender`. Every time you change one of these, you'll get a new _version_ of `CityVisitRecommender`.
 
-While we build the native integration with the Gemini API native python package, you can easily integrate Weave with the Gemini API yourself simply by initializing Weave with `weave.init('<your-project-name>')` and then wrapping the calls that call your LLMs with `weave.op()`. See our guide on [tracing](/guides/tracking/tracing) for more details.
+```python
+import os
+import google.generativeai as genai
+import weave
+
+
+class CityVisitRecommender(weave.Model):
+    model: str
+
+    @weave.op()
+    def predict(self, city: str) -> str:
+        model = genai.GenerativeModel(
+            model_name=self.model,
+            system_instruction="You are a helpful assistant meant to suggest all budget-friendly places to visit in a city",
+        )
+        response = model.generate_content(city)
+        return response.text
+
+
+weave.init(project_name="google_ai_studio-test")
+genai.configure(api_key=os.environ["GOOGLE_API_KEY"])
+city_recommender = CityVisitRecommender(model="gemini-1.5-flash")
+print(city_recommender.predict("New York"))
+print(city_recommender.predict("San Francisco"))
+print(city_recommender.predict("Los Angeles"))
+```
+
+### Serving a Weave Model
+
+Given a weave reference to any `weave.Model` object, you can spin up a fastapi server and [serve](https://wandb.github.io/weave/guides/tools/serve) it. You can serve your model by using the following command in the terminal:
+
+```shell
+weave serve weave:///your_entity/project-name/YourModel:<hash>
+```
+
+## Vertex API
+
+Full Weave support for the `Vertex AI SDK` python package is currently in development, however there is a way you can integrate Weave with the Vertex API. 
+
+Vertex API supports OpenAI SDK compatibility ([docs](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/call-gemini-using-openai-library)), and if this is a way you build your application, Weave will automatically track your LLM calls via our [OpenAI](/guides/integrations/openai) SDK integration.
+
+\* Please note that some features may not fully work as Vertex API doesn't implement the full OpenAI SDK capabilities.

docs/docs/guides/integrations/groq.md

@@ -86,7 +86,7 @@ Organizing experimentation is difficult when there are many moving pieces. By us
 
 In addition to versioning code and capturing inputs/outputs, [`Model`](../core-types/models)s capture structured parameters that control your application’s behavior, making it easy to find what parameters worked best. You can also use Weave Models with `serve`, and [`Evaluation`](../core-types/evaluations.md)s.
 
-In the example below, you can experiment with `WeaveModel`. Every time you change one of these, you'll get a new _version_ of `WeaveModel`.
+In the example below, you can experiment with `GroqCityVisitRecommender`. Every time you change one of these, you'll get a new _version_ of `GroqCityVisitRecommender`.
 
 ```python
 import os
@@ -129,7 +129,7 @@ print(city_recommender.predict("Los Angeles"))
 
 ### Serving a Weave Model
 
-Given a weave reference any WeaveModel object, you can spin up a fastapi server and [serve](https://wandb.github.io/weave/guides/tools/serve) it.
+Given a weave reference to any `weave.Model` object, you can spin up a fastapi server and [serve](https://wandb.github.io/weave/guides/tools/serve) it.
 
 | [![dspy_weave_model_serve.png](./imgs/groq/groq_weave_model_version.png)](https://wandb.ai/geekyrakshit/groq-test/weave/objects/GroqCityVisitRecommender/versions/6O1xPTJ9yFx8uuCjJAlI7KgcVYxXKn7JxfmVD9AQT5Q) |
 |---|

docs/docs/tutorial-eval.md

@@ -7,7 +7,7 @@ To iterate on an application, we need a way to evaluate if it's improving. To do
 ## 1. Build a `Model`
 
 `Model`s store and version information about your system, such as prompts, temperatures, and more.
-Weave automatically captures when they are used and update the version when there are changes.
+Weave automatically captures when they are used and updates the version when there are changes.
 
 `Model`s are declared by subclassing `Model` and implementing a `predict` function definition, which takes one example and returns the response.
 
@@ -185,7 +185,7 @@ examples = [
 # If you have already published the Dataset, you can run:
 # dataset = weave.ref('example_labels').get()
 
-# We define a scoring functions to compare our model predictions with a ground truth label.
+# We define a scoring function to compare our model predictions with a ground truth label.
 @weave.op()
 def fruit_name_score(target: dict, model_output: dict) -> dict:
     return {'correct': target['fruit'] == model_output['fruit']}