AI Agents

AI Agents is the dashboard for the AI side of your application: every model call, every tool an agent runs, the tokens they consume, what that costs, and how evaluators score the answers. It reads standard OpenTelemetry GenAI span attributes, so it works with Unhandled's own instrumentations and with any SDK that emits them. Open it from AI Agents in a project's sidebar (/o/:org/projects/:project/ai-agents).

Send AI telemetry

The quickest path is an instrumentation package:

• Vercel AI SDK wraps generateText, streamText, generateObject, streamObject, embed, embedMany and rerank.
• Mastra converts Mastra's agent, model and tool spans and their scorer results.

Any other span counts as AI as soon as it carries gen_ai.operation.name, gen_ai.provider.name (or the older gen_ai.system) or OpenLLMetry's llm.request.type, so OpenLLMetry and opentelemetry-instrumentation-openai-v2 spans work as they are. Older attribute names are renamed at ingest; the list is in semantic conventions.

Spans with the operation execute_tool are tool calls. Model calls are the spans with any other operation (chat, generate_content, embeddings and so on). Agent and workflow spans (invoke_agent, create_agent, invoke_workflow) usually repeat the total of the model calls inside them, so they are not counted again: an agent span counts as a call, with its tokens and cost, only when no model call that reported usage started inside it. A model call that runs before or after the agent span, in the same trace, does not stop it from counting.

Prompts and responses

The Vercel AI SDK and Mastra instrumentations record no prompts, responses or tool payloads unless you pass recordInputs: true and recordOutputs: true. Token counts, models, durations and costs arrive either way. Without content, conversation steps show no response text.

When content does arrive, ingest masks it before storage with the secret rules it applies to exceptions, plus a rule for email addresses. On a self-hosted server, INGEST_AI_SCRUB_DISABLE turns rules off, as a comma-separated list of names such as email,card. INGEST_AI_SCRUB_PATTERNS adds patterns, as a JSON array of regular expressions.

To remove stored content from a project's AI spans, run this on the server. --from and --to are optional ISO 8601 times that limit the range:

node ace ai:erase-content --project 12 --from 2026-10-01T00:00:00Z --to 2026-10-08T00:00:00Z

Deleting a project erases the content of its AI spans too.

Attributes

A call is an error when its span status is error. Total tokens are always input plus output; Unhandled does not read a total from the span.

Overview

Four cards summarize the range: AI calls, Token usage, Estimated cost (with what the cache saved) and Error rate. Charts show successful and failed calls, average and p95 response time, and token usage and estimated cost over time, both stacked by model. Token breakdown lists input, output, cached, cache write and reasoning tokens with the cache hit rate (cached input divided by all input) and cache savings.

Models

One row per model with calls, average and p95 latency, input, output and cached tokens, cost and errors. The header shows the number of models, total calls, total cost and the most used model. The Cost by model and Tokens by model charts stack the five busiest models and group the rest as Other.

Tools

One row per gen_ai.tool.name, up to 200 tools, with calls, success rate, errors, average and p95 duration and when the tool last ran. The cards show the number of tools, tool calls, success rate and failed calls.

Scorers

Scorers collects evaluation results. A result is a span event named gen_ai.evaluation.result, on any span in the project. Unhandled reads these event attributes:

Each scorer shows its result count, average score, a 10-bin distribution of scores between 0 and 1, counts per source and per label, and a score trend. The cards show scorers, results, average score and pass rate, which counts only results that have a label. Latest results lists the 50 newest with their explanations. When a result's target span belongs to a conversation, it links there.

To record a result by hand, add the event to any span:

import { trace } from '@opentelemetry/api'

trace.getActiveSpan()?.addEvent('gen_ai.evaluation.result', {
  'gen_ai.evaluation.name': 'refund_policy',
  'gen_ai.evaluation.score.value': 0.82,
  'gen_ai.evaluation.score.label': 'pass',
  'gen_ai.evaluation.explanation': 'Quotes the 30-day refund window.',
  'gen_ai.evaluation.source': 'live',
})

Conversations

Spans that share a gen_ai.conversation.id form a conversation. The tab lists the 100 most recent in the range with agent, models, messages (LLM calls), tool calls, tokens, cost, duration, errors and average score. Spans without the attribute do not appear here.

With the Vercel AI SDK instrumentation, wrap the work in withConversationId and every span started inside gets the attribute:

import { generateText } from 'ai'
import { openai } from '@ai-sdk/openai'
import { withConversationId } from '@unhandled/instrumentation-vercel-ai'

const reply = await withConversationId(`support-ticket-${ticket.id}`, () =>
  generateText({ model: openai('gpt-4o-mini'), prompt: ticket.question })
)

A conversation's page replays the agent loop step by step, up to 500 steps: for each step its offset, operation, model or tool, function ID, streaming flag, the first 500 characters of the response, input, output and cached tokens, cost, duration and status. Agent steps whose usage is already counted by their model calls show no tokens or cost. Evaluation results that target one of its spans are listed alongside, and a Traces panel lists every trace ID the conversation touched, ready to copy. The page reads the conversation's spans regardless of the selected time range.

Cost estimates

Unhandled prices usage when you open the page, from a table of published list prices in USD per million tokens, each with the date it applies from. Because nothing is priced at ingest, a model added to the table later is priced for its earlier calls too, and a price change applies only from its date.

• Uncached input tokens (input minus cache reads and cache writes) are billed at the input rate, cache reads at the cache-read rate, cache writes at the cache-write rate (the 5-minute rate for Anthropic), and output, reasoning included, at the output rate.
• Model ids match the model they name whatever the provider prefix or date suffix: anthropic.claude-sonnet-4-6-v1:0 (Bedrock), anthropic/claude-sonnet-4.6 (OpenRouter), models/gemini-2.5-flash and gpt-4o-mini-2024-07-18 all find their price. The table covers current Claude, OpenAI, Gemini and Mistral models.
• A span that carries gen_ai.usage.cost keeps that cost, and the Models tab marks the model Reported cost when every call did.
• A model without a price shows No price on the Models tab and counts as unpriced on the overview instead of adding $0.

In a trace

AI spans also appear in traces. In the span panel, every gen_ai.* attribute is grouped under AI, so prompts, responses and token counts sit together.