Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: Add traces and generations #10455

Merged
merged 2 commits into from
Jan 24, 2025
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion contents/docs/ai-engineering/dashboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ The [LLM observability dashboard](https://us.posthog.com/llm-observability) prov
classes="rounded"
/>

It can be filtered like any dashboard in PostHog, including by event, person, and group properties. Our observability SDKs autocapture especially useful properties like provider, tokens, cost, model, and more.

This dashboard is a great starting point for understanding your LLM usage and performance. You can use it to answer questions like:

- Are users using our LLM-powered features?
Expand All @@ -29,4 +31,4 @@ This dashboard is a great starting point for understanding your LLM usage and pe
- How many of my users are interacting with my LLM features?
- Are there generation latency spikes?

To dive into specific generation events, click on the [generations tab](https://us.posthog.com/llm-observability/generations) to get a list of all the recent generation events captured by PostHog.
To dive into specific generation events, click on the [generations](https://us.posthog.com/llm-observability/generations) or [traces](https://us.posthog.com/llm-observability/traces) tabs to get a list of each captured by PostHog.
2 changes: 1 addition & 1 deletion contents/docs/ai-engineering/llm-insights.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: LLM insights (beta)
title: LLM observability integrations (beta)
availability:
free: full
selfServe: full
Expand Down
35 changes: 35 additions & 0 deletions contents/docs/ai-engineering/traces-generations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
---
title: LLM traces and generations (beta)
availability:
free: full
selfServe: full
enterprise: full
---

Once you install PostHog's LLM observabilty SDK, it autocaptures LLM generations and traces. You can then view these in PostHog.

## Generations

Generations are an event that capture an LLM request. The [generations tab](https://us.posthog.com/llm-observability/generations) lists them along with the properties autocaptured by the PostHog like the person, model, total cost, token usage, and more.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something is a little off with this sentence:

"Generations are an event that capture an LLM request."

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also looks like this line should say "autocaptured by PostHog" instead of "autocaptured by the PostHog"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're right 😢


When you expand a generation, it includes the properties and metadata that every event has along with a conversation history with the role (system, user, assistant) as well as input and output content.

<ProductScreenshot
imageLight="https://res.cloudinary.com/dmukukwp6/image/upload/Clean_Shot_2025_01_23_at_14_47_43_2x_9223a3b730.png"
imageDark="https://res.cloudinary.com/dmukukwp6/image/upload/Clean_Shot_2025_01_23_at_14_47_17_2x_0cc9075dd5.png"
alt="LLM generations"
classes="rounded"
/>

## Traces

Traces are a collection of generations that capture a full interaction between a user and an LLM. The [traces tab](https://us.posthog.com/llm-observability/traces) lists them along with the properties autocaptured by the PostHog like the person, total cost, total latency, and more.

Clicking on a trace opens a timeline of the interaction with all the generation events enabling you to see the entire conversation, details about the trace, and the individual generation events.

<ProductScreenshot
imageLight="https://res.cloudinary.com/dmukukwp6/image/upload/Clean_Shot_2025_01_23_at_14_56_05_2x_5704304722.png"
imageDark="https://res.cloudinary.com/dmukukwp6/image/upload/Clean_Shot_2025_01_23_at_14_56_22_2x_705d03500f.png"
alt="LLM generations"
classes="rounded"
/>
66 changes: 26 additions & 40 deletions src/navs/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -2262,16 +2262,6 @@ export const docsMenu = {
{
name: 'Tools',
},
{
name: 'LLM insights',
url: '/docs/ai-engineering/llms',
icon: 'IconAIText',
color: 'yellow',
badge: {
title: 'Beta',
className: 'uppercase !bg-blue/10 !text-blue !dark:text-white !dark:bg-blue/50',
},
},
{
name: 'Autocapture',
url: '/docs/product-analytics/autocapture',
Expand Down Expand Up @@ -3232,46 +3222,42 @@ export const docsMenu = {
icon: 'IconDashboard',
color: 'red',
},
{
name: 'Traces and generations',
url: '/docs/ai-engineering/traces-generations',
icon: 'yellow',
color: 'orange',
},
{
name: 'Tutorials and guides',
url: '/docs/ai-engineering/tutorials',
icon: 'IconGraduationCap',
color: 'yellow',
color: 'seagreen',
featured: true,
},
{
name: 'Integrations',
},
{
name: 'LLM insights',
name: 'Observability integrations',
url: '/docs/ai-engineering/llm-insights',
icon: 'IconAIText',
color: 'blue',
featured: true,
},
{
name: 'Helicone',
url: '/docs/ai-engineering/helicone-posthog',
icon: 'IconWrench',
color: 'green',
},
{
name: 'Keywords AI',
url: '/docs/ai-engineering/keywords-ai-posthog',
icon: 'IconWrench',
color: 'blue',
},
{
name: 'Langfuse',
url: '/docs/ai-engineering/langfuse-posthog',
icon: 'IconWrench',
color: 'red',
},
{
name: 'Traceloop',
url: '/docs/ai-engineering/traceloop-posthog',
icon: 'IconWrench',
color: 'purple',
children: [
{
name: 'Helicone',
url: '/docs/ai-engineering/helicone-posthog',
},
{
name: 'Langfuse',
url: '/docs/ai-engineering/langfuse-posthog',
},
{
name: 'Traceloop',
url: '/docs/ai-engineering/traceloop-posthog',
},
{
name: 'Keywords AI',
url: '/docs/ai-engineering/keywords-ai-posthog',
},
],
},
],
},
Expand Down
Loading