Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blog optimization [in progress] #3100

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions bifrost/app/blog/blogs/ai-agent-builders/metadata.json
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
{
"title": "6 Awesome Platforms & Frameworks for Building AI Agents (Open-Source & More)",
"title1": "6 Awesome Platforms & Frameworks for Building AI Agents (Open-Source & More)",
"title2": "6 Awesome Platforms & Frameworks for Building AI Agents (Open-Source & More)",
"description": "Today, we are covering 6 of our favorite platforms for building AI agents — whether you need complex multi-agent systems or a simple no-code solution. ",
"title": "7 Awesome Platforms & Frameworks for Building AI Agents (Open-Source & More)",
"title1": "7 Awesome Platforms & Frameworks for Building AI Agents (Open-Source & More)",
"title2": "7 Awesome Platforms & Frameworks for Building AI Agents (Open-Source & More)",
"description": "Today, we are covering 7 of our favorite platforms for building AI agents — whether you need complex multi-agent systems or a simple no-code solution. ",
"images": "/static/blog/ai-agent-builders/cover.webp",
"time": "12 minute read",
"author": "Lina Lam",
Expand Down
157 changes: 115 additions & 42 deletions bifrost/app/blog/blogs/ai-agent-builders/src.mdx

Large diffs are not rendered by default.

10 changes: 3 additions & 7 deletions bifrost/app/blog/blogs/ai-best-practices/src.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -124,16 +124,12 @@ LLMs can be manipulated into convincing the user to input sensitive information,

**<span style={{color: '#0ea5e9'}}>How Helicone can help you:</span>**

Helicone provides <a href="https://docs.helicone.ai/features/advanced-usage/moderations" target="_blank" rel="noopener noreferrer">moderation</a> and <a href="https://docs.helicone.ai/features/advanced-usage/llm-security" target="_blank" rel="noopener noreferrer">LLM security</a> features to help you check whether the user message is potentially harmful, and enhance OpenAI chat completions with automated security checks, which include user messages for threads, block injection threats and threat details back to you.
Helicone provides <a href="https://docs.helicone.ai/features/advanced-usage/moderations" target="_blank" rel="noopener">moderation</a> and <a href="https://docs.helicone.ai/features/advanced-usage/llm-security" target="_blank" rel="noopener">LLM security</a> features to help you check whether the user message is potentially harmful, and enhance OpenAI chat completions with automated security checks, which include user messages for threads, block injection threats and threat details back to you.

---

### Conclusion
## Bottom Line

Keeping your AI app reliable hinges on effective observability and performance monitoring. This means defining important performance metrics, setting up thorough logging, monitoring your outputs regularly, and ensuring safety and security measures are in place. By following these best practices, you can boost the performance and reliability of your LLM deployments and accelerate your AI development.

---

### Try Helicone for free.

<a href="https://www.helicone.ai/" target="_blank" rel="noopener noreferrer">Sign up</a> or <a href="https://www.helicone.ai/contact" target="_blank" rel="noopener noreferrer">contact us</a>.
<Questions />
2 changes: 2 additions & 0 deletions bifrost/app/blog/blogs/ai-safety/src.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,5 @@ Last but not least, we are actively developing Drift Detection. Understanding th
## Ethical Observability as a Cornerstone for the Future

In a world increasingly dependent on AI, it's crucial that we deploy these powerful technologies responsibly and ethically. Helicone addresses this need by providing not just robust performance tracking but also specialized features for ethical observability, including Two-Way Door Auditing, Segmentation and ETL, and Drift Detection. These features serve as ethical cornerstones, ensuring that AI systems adhere to legal standards and societal values. As we navigate the complexities of AI integration into various sectors, tools like Helicone become not just an operational necessity but a social imperative, empowering organizations to deploy AI both efficiently and ethically.

<Questions />
34 changes: 14 additions & 20 deletions bifrost/app/blog/blogs/autoGPT/src.mdx
Original file line number Diff line number Diff line change
@@ -1,12 +1,11 @@

AutoGPT is diligently developing their [Auto-GPT-Benchmarks repository](https://github.com/Significant-Gravitas/Auto-GPT-Benchmarks). Their goal? To construct the optimal evaluation pipeline for comparing different agents.
AutoGPT is diligently developing their <a href="https://github.com/Significant-Gravitas/Auto-GPT-Benchmarks" target="_blank" rel="noopener nofollow">Auto-GPT-Benchmarks repository</a>. Their goal? To construct the optimal evaluation pipeline for comparing different agents.

![AutoGPT x Helicone: Optimizing Evaluation Pipelines](/static/blog/autogpt.webp)

AutoGPT is fully leveraging the capabilities of Helicone without modifying a single line of code. Here are the key features that facilitate this synergy:

- **Proxy Integration:** Helicone's role as a proxy allows AutoGPT to maintain their codebase intact. Learn more about this feature in our [MITM Proxy documentation](https://docs.helicone.ai/tools/mitm-proxy).
- **Caching:** For minor code modifications that don't necessitate re-calling the LLM for an entire CI pipeline, requests can be cached on edge servers. This feature saves AutoGPT over $10 per PR! You can read more about this in our [Caching documentation](https://docs.helicone.ai/features/advanced-usage/caching).
- **Proxy Integration:** Helicone's role as a proxy allows AutoGPT to maintain their codebase intact. Learn more about this feature in our <a href="https://docs.helicone.ai/tools/mitm-proxy" target="_blank" rel="noopener">MITM Proxy documentation</a>.
- **Caching:** For minor code modifications that don't necessitate re-calling the LLM for an entire CI pipeline, requests can be cached on edge servers. This feature saves AutoGPT over $10 per PR! You can read more about this in our <a href="https://docs.helicone.ai/features/advanced-usage/caching" target="_blank" rel="noopener">Caching documentation</a>.
- **GraphQL:** Our data extraction API enables AutoGPT to generate custom reports upon the completion of a CI job.

## AutoGPT's Workflow with Helicone
Expand All @@ -22,9 +21,7 @@ bash -c "$(curl -fsSL https://raw.githubusercontent.com/Helicone/helicone/main/m

```

Within the benchmarks, AutoGPT implemented a Python library where they could set specific custom properties for detailed measurements, as shown here. You can learn more about this in our [Custom Properties documentation](https://docs.helicone.ai/features/advanced-usage/custom-properties).


Within the benchmarks, AutoGPT implemented a Python library where they could set specific custom properties for detailed measurements, as shown here. You can learn more about this in our <a href="https://docs.helicone.ai/features/advanced-usage/custom-properties" target="_blank" rel="noopener">Custom Properties documentation</a>.

```python
HeliconeLockManager.write_custom_property("job_id", "1")
Expand All @@ -39,41 +36,36 @@ export HELICONE_CACHE_ENABLED="true"

The total integration process required at most 5 lines of code, which enabled AutoGPT to immediately get rich dashboards and save costs on their CI jobs.


### Data Ingest

AutoGPT can track how different agents are impacting their costs.

![Agent Comparisons](/static/blog/agentComparisons.webp)
*AutoGPT's agent comparison dashboard*
_AutoGPT's agent comparison dashboard_

If they wish to examine specific requests, they can do this by using the filter feature.

![Agent Filters](/static/blog/agentFilters.webp)
*Filtering feature for examining specific requests*

_Filtering feature for examining specific requests_

### Determining Cost Savings

For scenarios where testing the agent's functionality is needed but calling the API is not, such as small code changes, they can monitor their cache usage effortlessly through the dashboards. Here's an example:


![Cache Page Stats](/static/blog/cachePageStats.webp)
*Dashboard showing cache usage statistics*
_Dashboard showing cache usage statistics_

We also maintain a log of each cached request, ensuring that caching is effective and marking agents as "Cacheable Agents".

![Cache Request Table](/static/blog/cacheRequestTable.webp)
*Log of each cached request*

_Log of each cached request_

## The Road Ahead

We are currently developing a suite of GraphQL endpoints that will allow AutoGPT to easily ingest some of their data and add it directly to the reports after a run.

![GraphQL](/static/blog/graphQL.webp)
*GraphQL endpoints in development*

_GraphQL endpoints in development_

This development is being paired with deep links so that we can have a tight integration between report generation and Helicone. Here is a preview of what a benchmark report will look like:

Expand All @@ -87,7 +79,7 @@ Challenge go-to-market
Number of OpenAI calls: 231
Total Cache hits: 231
Cache $ saved: $231
Link: https://helicone.ai/requests?propertyFilters=%5B%7B%22key%22%3A%22challenge%22%2C%22value%22%3A%22got-to-market%22%7D%5D
Link: https://helicone.ai/requests?propertyFilters=%5B%7B%22key%22%3A%22challenge%22%2C%22value%22%3A%22got-to-market%22%7D%5D

Model breakdown
| | gpt4 | claude | gpt3.5 |
Expand All @@ -103,7 +95,7 @@ Challenge send-email
Number of OpenAI calls: 231
Total Cache hits: 231
Cache $ saved: $231
Link: https://helicone.ai/requests?propertyFilters=%5B%7B%22key%22%3A%22challenge%22%2C%22value%22%3A%22send-email%22%7D%5D
Link: https://helicone.ai/requests?propertyFilters=%5B%7B%22key%22%3A%22challenge%22%2C%22value%22%3A%22send-email%22%7D%5D

Model breakdown
| | gpt4 | claude | gpt3.5 |
Expand All @@ -125,4 +117,6 @@ Challenge send-email

## Thank You for Reading!

We appreciate your time in reading our first blog post. We are excited to be partnering with AutoGPT to enable rich logging for them and deliver value using Helicone. If you are interested in learning more about Helicone or would like to meet the team, please email me at [email protected] or join our discord!
We appreciate your time in reading our first blog post. We are excited to be partnering with AutoGPT to enable rich logging for them and deliver value using Helicone. If you are interested in learning more about Helicone or would like to meet the team, please email me at [email protected] or join our discord!

<Questions />
105 changes: 63 additions & 42 deletions bifrost/app/blog/blogs/best-arize-alternatives/src.mdx
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
![Arize AI vs. Helicone, which one is better?](/static/blog/arize-alternatives/helicone-vs-arize.webp)

### Introduction
## Introduction

As the adoption of Large Language Models (LLMs) continues to grow, the need for robust observability tools has become paramount. These tools help developers and data scientists monitor, analyze, and optimize their LLM applications. In this comparison, we'll explore two leading platforms in the LLM observability space: Helicone and Arize Phoenix. Both offer unique features and capabilities, but choosing the right tool can significantly impact your AI development workflow.

### Overview: Helicone vs. Arize Phoenix
## Overview: Helicone vs. Arize Phoenix

| Feature | Helicone | Arize Phoenix |
| -------------------- | -------- | ------------- |
Expand All @@ -24,45 +24,13 @@ As the adoption of Large Language Models (LLMs) continues to grow, the need for
| User Tracking | ✅ | ❌ |
| User Feedback | ✅ | ❌ |

---

### Use Case Scenarios

Different tools excel in different scenarios. Here's a quick guide to help you choose the right tool for your specific needs:

1. **Small Startup with Limited Budget**

- Best Tool: Helicone
- Why: Offers a free tier and flexible pricing, making it accessible for startups

2. **Large Enterprise with Complex Workflows**

- Best Tool: Helicone
- Why: Robust evaluation capabilities and scalability for enterprise-level needs

3. **Research Team Focused on Experimentation**

- Best Tool: Helicone
- Why: Comprehensive experiment features and prompt management

4. **Solo Developer Working on Side Projects**

- Best Tool: Helicone
- Why: Easy integration and user-friendly interface

5. **AI-Focused Company with High Volume LLM Usage**
- Best Tool: Helicone
- Why: Advanced caching, cost analysis, and scalability features

---

# **1. Helicone**
## 1. Helicone

**Designed for: developers & analysts**

![Helicone Dashboard Image](/static/blog/arize-alternatives/helicone-dashboard.webp)

## What is Helicone?
### What is Helicone?

Helicone is a comprehensive LLM observability platform designed for developers of all skill levels. It offers a wide range of features including request logging, caching, prompt management, and advanced analytics. With its open-source nature and self-hosting options, Helicone provides flexibility and control over your data.

Expand All @@ -74,19 +42,19 @@ Helicone is a comprehensive LLM observability platform designed for developers o
4. **User Tracking** - Gain insights into user interactions and behaviors within your LLM-powered applications.
5. **Cost Analysis** - Monitor and optimize your LLM usage costs with detailed analytics.

## How does Helicone compare to Arize Phoenix?
### How does Helicone compare to Arize Phoenix?

While both tools offer strong observability features, Helicone stands out with its user-friendly approach and comprehensive feature set. Unlike Arize Phoenix, Helicone provides self-hosting options, user tracking, and user feedback collection. Its flexible pricing model and free tier make it more accessible for smaller teams and individual developers.

---

# **2. Arize Phoenix**
## 2. Arize Phoenix

**Designed for: Data scientists & ML engineers**

![Arize Phoenix Dashboard Image](/static/blog/arize-alternatives/arize-ai-dashboard.webp)

## What is Arize Phoenix?
### What is Arize Phoenix?

Arize Phoenix is an open-source LLM observability tool that focuses on providing robust evaluation and monitoring capabilities for LLM applications. It offers features like tracing, prompt management, and performance analytics, making it suitable for data scientists and ML engineers working on complex AI projects.

Expand All @@ -96,23 +64,74 @@ Arize Phoenix is an open-source LLM observability tool that focuses on providing
2. **Evaluation** - Comprehensive tools for assessing LLM performance and output quality.
3. **Agent Tracing** - Visualize and analyze multi-step LLM interactions and workflows.

## How does Arize Phoenix compare to Helicone?
### How does Arize Phoenix compare to Helicone?

Arize Phoenix excels in its evaluation capabilities and is well-suited for data scientists and ML engineers working on complex LLM projects. However, it lacks some of the developer-friendly features that Helicone offers, such as self-hosting options, user tracking, and user feedback collection. Arize Phoenix's pricing model may also be less flexible compared to Helicone's tiered approach.

---

### So, which LLM observability tool suits you better?
## Arize vs. Helicone: which LLM observability tool suits you better?

Different tools excel in different scenarios. Here's a quick guide to help you choose the right tool for your specific needs:

1. **Small Startup with Limited Budget**

- Best Tool: Helicone
- Why: Offers a free tier and flexible pricing, making it accessible for startups

2. **Large Enterprise with Complex Workflows**

- Best Tool: Helicone
- Why: Robust evaluation capabilities and scalability for enterprise-level needs

3. **Research Team Focused on Experimentation**

- Best Tool: Helicone
- Why: Comprehensive experiment features and prompt management

4. **Solo Developer Working on Side Projects**

- Best Tool: Helicone
- Why: Easy integration and user-friendly interface

5. **AI-Focused Company with High Volume LLM Usage**
- Best Tool: Helicone
- Why: Advanced caching, cost analysis, and scalability features

---

## Bottom Line

Both Helicone and Arize Phoenix offer powerful features for LLM observability, but they cater to slightly different audiences. Helicone's user-friendly approach, comprehensive feature set, and flexible pricing make it an excellent choice for a wide range of users, from solo developers to small and medium-sized teams. Its self-hosting options and advanced features like user tracking and feedback collection give it an edge in many scenarios.

Arize Phoenix, on the other hand, shines in its evaluation capabilities and may be preferred by data scientists and ML engineers working on complex LLM projects. However, its lack of self-hosting options and more specialized focus might make it less suitable for smaller teams or individual developers.

Ultimately, the choice between Helicone and Arize Phoenix depends on your specific needs, team size, and the complexity of your LLM applications. For most users, especially those looking for an all-in-one solution with a gentle learning curve, Helicone appears to be the more versatile and accessible option.

<CallToAction
title="Ready to scale your LLM app?"
description="Track your LLM usage, optimize costs, improve your prompts, and scale your LLM app with Helicone."
primaryButtonText="Try Helicone for free"
primaryButtonLink="https://www.helicone.ai/signup"
secondaryButtonText="Contact us"
secondaryButtonLink="https://www.helicone.ai/contact"
/>

### You might be interested in

- <a href="/blog/langsmith-alternatives" rel="noopener" target="_blank">
Comparing Langsmith vs Helicone
</a>
- <a href="/blog/braintrust-alternatives" rel="noopener" target="_blank">
Comparing Braintrust vs Helicone
</a>
- <a href="/blog/best-langfuse-alternatives" rel="noopener" target="_blank">
Comparing Langfuse vs Helicone
</a>

---

### Frequently Asked Questions
## Frequently Asked Questions

1. **Q: What is the main difference between Helicone and Arize Phoenix?**
A: The main difference lies in their target audience and feature set. Helicone is more developer-friendly with features like self-hosting and user tracking, while Arize Phoenix focuses on robust evaluation tools for data scientists and ML engineers.
Expand All @@ -128,3 +147,5 @@ Ultimately, the choice between Helicone and Arize Phoenix depends on your specif

5. **Q: How do these tools handle data privacy and security?**
A: Both tools take data privacy seriously. Helicone offers self-hosting options for complete data control, while Arize Phoenix, being open-source, allows for scrutiny of its security practices. Always review the latest security features and compliance certifications when making your decision.

<Questions />
Loading
Loading