test

docs/notebooks/online_monitoring.ipynb

@@ -4,14 +4,13 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "<!-- docusaurus_head_meta::start\n",
-    "---\n",
-    "title: Integrating with Weave - Production Dashboard\n",
-    "---\n",
+    "## <!-- docusaurus_head_meta::start\n",
+    "\n",
+    "## title: Integrating with Weave - Production Dashboard\n",
+    "\n",
     "docusaurus_head_meta::end -->\n",
     "\n",
-
-    "<!--- @wandbcode{cod-notebook} -->"
+    "<!--- @wandbcode{cod-notebook} -->\n"
    ]
   },
   {
@@ -23,26 +22,28 @@
     "The GenAI tooling landscape is rapidly evolving - new frameworks, tools, and applications are emerging all the time. Weave aims to be a one-stop-shop for all your GenAI monitoring and evaluation needs. This also means that sometimes it is necessary to integrate with existing platforms or extend Weave to fit the specific needs of your project or organization.\n",
     "\n",
     "In this cookbook, we'll demonstrate how to leverage Weave's powerful APIs and functions to create a custom dashboard for production monitoring as an extension to the Traces view in Weave. We'll focus on:\n",
-    "* Fetching traces, costs, feedback, and other metrics from Weave\n",
-    "* Creating aggregate views for user feedback and cost distribution\n",
-    "* Creating visualizations for token usage and latency over time\n",
     "\n",
-    "You can try out the dashboard with your own Weave project by installing streamlit and running [this production dashboard script](https://github.com/NiWaRe/agent-dev-collection)!"
+    "- Fetching traces, costs, feedback, and other metrics from Weave\n",
+    "- Creating aggregate views for user feedback and cost distribution\n",
+    "- Creating visualizations for token usage and latency over time\n",
+    "\n",
+    "You can try out the dashboard with your own Weave project by installing streamlit and running [this production dashboard script](https://github.com/NiWaRe/agent-dev-collection)!\n"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "<img src=\"https://github.com/NiWaRe/knowledge-worker-weave/blob/master/screenshots/dashboard_weave_preview.jpg?raw=true\" width=\"1000\" alt=\"Example Production Dashboard with Weave\" />"
+    "<img src=\"https://github.com/NiWaRe/knowledge-worker-weave/blob/master/screenshots/dashboard_weave_preview.jpg?raw=true\" width=\"1000\" alt=\"Example Production Dashboard with Weave\" />\n"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "# 1. Setup\n",
-    "To follow along this tutorial you'll only need to install the following packages:"
+    "\n",
+    "To follow along this tutorial you'll only need to install the following packages:\n"
    ]
   },
   {
@@ -58,18 +59,19 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# 2. Implementation"
+    "# 2. Implementation\n"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "## 2.1 Initializing Weave Client and Defining Costs\n",
-    "First, we'll set up a function to initialize the Weave client and add costs for each model. \n",
     "\n",
-    "* We have included the standard costs for many standard models but we also make it easy to add your own custom costs and custom models. In the following we'll show how to add custom costs for a few models and use the standard costs for the rest.\n",
-    "* The costs are calculate based on the tracked tokens for each call in Weave. For many LLM vendor libraries, we will automatically track the token usage, but it is also possible to return custom token counts for any call. See this cookbook on how to define the token count and cost calculation for a custom model - [custom cost cookbook](https://weave-docs.wandb.ai/reference/gen_notebooks/custom_model_cost#setting-up-a-model-with-weave)."
+    "First, we'll set up a function to initialize the Weave client and add costs for each model.\n",
+    "\n",
+    "- We have included the standard costs for many standard models but we also make it easy to add your own custom costs and custom models. In the following we'll show how to add custom costs for a few models and use the standard costs for the rest.\n",
+    "- The costs are calculate based on the tracked tokens for each call in Weave. For many LLM vendor libraries, we will automatically track the token usage, but it is also possible to return custom token counts for any call. See this cookbook on how to define the token count and cost calculation for a custom model - [custom cost cookbook](https://weave-docs.wandb.ai/reference/gen_notebooks/custom_model_cost#setting-up-a-model-with-weave).\n"
    ]
   },
   {
@@ -119,10 +121,11 @@
     "                prompt_token_cost=prompt_cost,\n",
     "                completion_token_cost=completion_cost,\n",
     "            )\n",
-    "        return client\n",
     "    except Exception as e:\n",
     "        print(f\"Failed to initialize Weave client for project '{project_name}': {e}\")\n",
     "        return None\n",
+    "    else:\n",
+    "        return client\n",
     "\n",
     "\n",
     "client = init_weave_client(PROJECT_NAME)"
@@ -133,19 +136,22 @@
    "metadata": {},
    "source": [
     "## 2.2 Fetching Calls Data from Weave\n",
+    "\n",
     "In order to fetch call data from Weave, we have two options:\n",
+    "\n",
     "1. Fetching Data call-by-call\n",
     "2. Using high-level APIs\n",
     "\n",
     "### 2.2.1 Fetching Data call-by-call\n",
+    "\n",
     "The first option to access data from Weave is to retrieve a list of filtered calls and extract the wanted data call-by-call. For that we can use the `calls_query_stream` API to fetch the calls data from Weave:\n",
     "\n",
-    "* `calls_query_stream` API: This API allows us to fetch the calls data from Weave.\n",
-    "* `filter` dictionary: This dictionary contains the filter parameters to fetch the calls data - see [here](https://weave-docs.wandb.ai/reference/python-sdk/weave/trace_server/weave.trace_server.trace_server_interface/#class-callschema) for more details.\n",
-    "* `expand_columns` list: This list contains the columns to expand in the calls data.\n",
-    "* `sort_by` list: This list contains the sorting parameters for the calls data.\n",
-    "* `include_costs` boolean: This boolean indicates whether to include the costs in the calls data.\n",
-    "* `include_feedback` boolean: This boolean indicates whether to include the feedback in the calls data."
+    "- `calls_query_stream` API: This API allows us to fetch the calls data from Weave.\n",
+    "- `filter` dictionary: This dictionary contains the filter parameters to fetch the calls data - see [here](https://weave-docs.wandb.ai/reference/python-sdk/weave/trace_server/weave.trace_server.trace_server_interface/#class-callschema) for more details.\n",
+    "- `expand_columns` list: This list contains the columns to expand in the calls data.\n",
+    "- `sort_by` list: This list contains the sorting parameters for the calls data.\n",
+    "- `include_costs` boolean: This boolean indicates whether to include the costs in the calls data.\n",
+    "- `include_feedback` boolean: This boolean indicates whether to include the feedback in the calls data.\n"
    ]
   },
   {
@@ -183,10 +189,11 @@
     "            itertools.islice(calls_stream, limit)\n",
     "        )  # limit the number of calls to fetch if too many\n",
     "        print(f\"Fetched {len(calls)} calls.\")\n",
-    "        return calls\n",
     "    except Exception as e:\n",
     "        print(f\"Error fetching calls: {e}\")\n",
     "        return []\n",
+    "    else:\n",
+    "        return calls\n",
     "\n",
     "\n",
     "calls = fetch_calls(client, PROJECT_NAME, datetime.now() - timedelta(days=1), True, 100)"
@@ -369,7 +376,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Processing the calls is very easy with the return from Weave - we'll extract the relevant information and store it in a list of dictionaries. We'll then convert the list of dictionaries to a pandas DataFrame and return it. "
+    "Processing the calls is very easy with the return from Weave - we'll extract the relevant information and store it in a list of dictionaries. We'll then convert the list of dictionaries to a pandas DataFrame and return it.\n"
    ]
   },
   {
@@ -536,8 +543,9 @@
    "metadata": {},
    "source": [
     "### 2.2.2 Using high-level APIs\n",
-    "Instead of goin through every call Weave also provides high-level APIs to directly access model costs, feedback, and other metrics. \n",
-    "For example, for the cost, we'll use the `query_costs` API to fetch the costs of all used LLMs using in project:"
+    "\n",
+    "Instead of goin through every call Weave also provides high-level APIs to directly access model costs, feedback, and other metrics.\n",
+    "For example, for the cost, we'll use the `query_costs` API to fetch the costs of all used LLMs using in project:\n"
    ]
   },
   {
@@ -732,7 +740,8 @@
    "metadata": {},
    "source": [
     "## 2.4 Gathering inputs and generating visualizations\n",
-    "Next, we can generate the visualizations using plotly. This is the most basic dashboard, but you can customize it as you like! For a more complex example, check out a Streamlit example [here](https://github.com/NiWaRe/knowledge-worker-weave/blob/master/prod_dashboard.py)."
+    "\n",
+    "Next, we can generate the visualizations using plotly. This is the most basic dashboard, but you can customize it as you like! For a more complex example, check out a Streamlit example [here](https://github.com/NiWaRe/knowledge-worker-weave/blob/master/prod_dashboard.py).\n"
    ]
   },
   {
@@ -2728,16 +2737,17 @@
    "metadata": {},
    "source": [
     "# Conclusion\n",
+    "\n",
     "In this cookbook, we demonstrated how to create a custom production monitoring dashboard using Weave's APIs and functions. Weave currently focuses on fast integrations for easy input of data as well as extraction of the data for custom processes.\n",
     "\n",
-    "* **Data Input:** \n",
-    "    * Framework-agnostic tracing with [@weave-op()](https://weave-docs.wandb.ai/quickstart#2-log-a-trace-to-a-new-project) decorator and the possibility to import calls from CSV (see related [import cookbook](https://weave-docs.wandb.ai/reference/gen_notebooks/import_from_csv))\n",
-    "    * Service API endpoints to log to Weave from for various programming frameworks and languages, see [here](https://weave-docs.wandb.ai/reference/service-api/call-start-call-start-post) for more details.\n",
-    "* **Data Output:**\n",
-    "    * Easy download of the data in CSV, TSV, JSONL, JSON formats - see [here](https://weave-docs.wandb.ai/guides/tracking/tracing#querying--exporting-calls) for more details.\n",
-    "    * Easy export using programmatic access to the data - see \"Use Python\" section in the export panel as described in this cookbook. See [here](https://weave-docs.wandb.ai/guides/tracking/tracing#querying--exporting-calls) for more details.\n",
+    "- **Data Input:**\n",
+    "  - Framework-agnostic tracing with [@weave-op()](https://weave-docs.wandb.ai/quickstart#2-log-a-trace-to-a-new-project) decorator and the possibility to import calls from CSV (see related [import cookbook](https://weave-docs.wandb.ai/reference/gen_notebooks/import_from_csv))\n",
+    "  - Service API endpoints to log to Weave from for various programming frameworks and languages, see [here](https://weave-docs.wandb.ai/reference/service-api/call-start-call-start-post) for more details.\n",
+    "- **Data Output:**\n",
+    "  - Easy download of the data in CSV, TSV, JSONL, JSON formats - see [here](https://weave-docs.wandb.ai/guides/tracking/tracing#querying--exporting-calls) for more details.\n",
+    "  - Easy export using programmatic access to the data - see \"Use Python\" section in the export panel as described in this cookbook. See [here](https://weave-docs.wandb.ai/guides/tracking/tracing#querying--exporting-calls) for more details.\n",
     "\n",
-    "This custom dashboard extends Weave's native Traces view, allowing for tailored monitoring of LLM applications in production. If you're interested in viewing a more complex dashboard, check out a Streamlit example where you can add your own Weave project URL [in this repo](https://github.com/NiWaRe/agent-dev-collection)."
+    "This custom dashboard extends Weave's native Traces view, allowing for tailored monitoring of LLM applications in production. If you're interested in viewing a more complex dashboard, check out a Streamlit example where you can add your own Weave project URL [in this repo](https://github.com/NiWaRe/agent-dev-collection).\n"
    ]
   }
  ],

examples/agents/programmer.py

@@ -45,9 +45,10 @@ def list_files(directory: str) -> str:
         if len(result) > LENGTH_LIMIT:
             result = result[:LENGTH_LIMIT]
             result += "\n... (truncated)"
-        return result
     except Exception as e:
         return json.dumps([str(e)])
+    else:
+        return result
 
 
 @weave.op()
@@ -64,9 +65,10 @@ def write_to_file(path: str, content: str) -> str:
     try:
         with open(path, "w") as f:
             f.write(content)
-        return "File written successfully."
     except Exception as e:
         return str(e)
+    else:
+        return "File written successfully."
 
 
 @weave.op()

pyproject.toml

@@ -169,6 +169,7 @@ select = [
   "UP",     # https://docs.astral.sh/ruff/rules/#pyupgrade-up
   "TRY002", # https://docs.astral.sh/ruff/rules/raise-vanilla-class/
   "TRY004", # https://docs.astral.sh/ruff/rules/type-check-without-type-error/
+  "TRY300", # https://docs.astral.sh/ruff/rules/try-consider-else/
 ]
 ignore = [
   # we use Google style

weave/integrations/langchain/langchain.py

@@ -355,9 +355,10 @@ def attempt_patch(self) -> bool:
             register_configure_hook(
                 weave_tracing_callback_var, True, WeaveTracer, "WEAVE_TRACE_LANGCHAIN"
             )
-            return True
         except Exception:
             return False
+        else:
+            return True
 
     def undo_patch(self) -> bool:
         if not hasattr(self, "original_trace_state"):
@@ -370,10 +371,10 @@ def undo_patch(self) -> bool:
             else:
                 os.environ["WEAVE_TRACE_LANGCHAIN"] = self.original_trace_state
             weave_tracing_callback_var.set(None)
-
-            return True
         except Exception:
             return False
+        else:
+            return True
 
 
 langchain_patcher = LangchainPatcher()

weave/integrations/llamaindex/llamaindex.py

@@ -174,9 +174,10 @@ def attempt_patch(self) -> bool:
             self._original_handler = llama_index.core.global_handler
 
             llama_index.core.global_handler = WeaveCallbackHandler()
-            return True
         except Exception:
             return False
+        else:
+            return True
 
     def undo_patch(self) -> bool:
         if not hasattr(self, "_original_handler"):
@@ -185,9 +186,10 @@ def undo_patch(self) -> bool:
             import llama_index.core
 
             llama_index.core.global_handler = self._original_handler
-            return True
         except Exception:
             return False
+        else:
+            return True
 
 
 llamaindex_patcher = LLamaIndexPatcher()

weave/scorers/json_scorer.py

@@ -12,6 +12,7 @@ class ValidJSONScorer(Scorer):
     def score(self, output: Any) -> dict:
         try:
             _ = json.loads(output)
-            return {"json_valid": True}
         except json.JSONDecodeError:
             return {"json_valid": False}
+        else:
+            return {"json_valid": True}

weave/scorers/pydantic_scorer.py

@@ -16,12 +16,14 @@ def score(self, output: Any) -> dict:
         if isinstance(output, str):
             try:
                 self.model.model_validate_json(output)
-                return {"valid_pydantic": True}
             except ValidationError:
                 return {"valid_pydantic": False}
+            else:
+                return {"valid_pydantic": True}
         else:
             try:
                 self.model.model_validate(output)
-                return {"valid_pydantic": True}
             except ValidationError:
                 return {"valid_pydantic": False}
+            else:
+                return {"valid_pydantic": True}

weave/scorers/string_scorer.py

@@ -26,11 +26,12 @@ def check_levenshtein(self) -> "LevenshteinScorer":
             from Levenshtein import distance
 
             self.distance = distance
-            return self
         except ImportError:
             raise ValueError(
                 "Levenshtein package not found. Please install it with `pip install Levenshtein`"
             )
+        else:
+            return self
 
     @weave.op
     def score(self, output: str, target: str) -> dict:

weave/scorers/xml_scorer.py

@@ -17,6 +17,7 @@ def score(self, output: Union[str, dict]) -> dict:
 
         try:
             ET.fromstring(xml_string)
-            return {"xml_valid": True}
         except ET.ParseError:
             return {"xml_valid": False}
+        else:
+            return {"xml_valid": True}

weave/trace/op_extensions/accumulator.py

@@ -120,13 +120,14 @@ def __next__(self) -> Generator[None, None, V]:
                 if get_raise_on_captured_errors():
                     raise
                 log_once(logger.error, ON_YIELD_MSG.format(traceback.format_exc()))
-            return value
         except (StopIteration, StopAsyncIteration) as e:
             self._call_on_close_once()
             raise
         except Exception as e:
             self._call_on_error_once(e)
             raise
+        else:
+            return value
 
     def __aiter__(self) -> "_IteratorWrapper":
         return self
@@ -158,13 +159,14 @@ async def __anext__(self) -> Generator[None, None, V]:
                 if get_raise_on_captured_errors():
                     raise
                 log_once(logger.error, ON_AYIELD_MSG.format(traceback.format_exc()))
-            return value
         except (StopAsyncIteration, StopIteration) as e:
             self._call_on_close_once()
             raise StopAsyncIteration
         except Exception as e:
             self._call_on_error_once(e)
             raise
+        else:
+            return value
 
     def __del__(self) -> None:
         self._call_on_close_once()

weave/trace/pypi_version_check.py

@@ -114,9 +114,10 @@ def wrapper(
             result = q.get(True, timeout)
             if isinstance(result, Exception):
                 raise result.with_traceback(sys.exc_info()[2])
-            return result, thread
         except queue.Empty:
             return None, thread
+        else:
+            return result, thread
 
     return wrapper
 

weave/trace/weave_client.py

@@ -1525,9 +1525,10 @@ def safe_current_wb_run_id() -> Optional[str]:
         wandb_run = wandb.run
         if wandb_run is None:
             return None
-        return f"{wandb_run.entity}/{wandb_run.project}/{wandb_run.id}"
     except ImportError:
         return None
+    else:
+        return f"{wandb_run.entity}/{wandb_run.project}/{wandb_run.id}"
 
 
 def check_wandb_run_matches(