docs

Semantic conventions

The span kinds, attribute names and unhandled.* keys that decide how Unhandled classifies, groups and displays your telemetry.

unhandled stores every attribute you send, but its views read specific names. A span becomes a request, a query or a job because of its kind and a few attributes, following the OpenTelemetry semantic conventions plus a small set of unhandled.* keys. Use this page when you instrument code by hand, write your own instrumentation, or wonder why a span lands in the wrong place. The rules live in @unhandled/protocol, shared by the agents and the ingest server.

Span categories

Each span gets one category. The rules run in this order and the first match wins:

OrderCategoryRule
1httpKind SERVER and http.request.method is set
2queryKind CLIENT, db.system.name is set and is not redis
3externalKind CLIENT and http.request.method is set
4cachecache.operation is set
5mcpmcp.method.name is set
6jobmessaging.system is set
7clientry_point.type or unhandled.entry_point is cli, or kind INTERNAL and process.executable.name is set
8aigen_ai.operation.name, gen_ai.provider.name (or the older gen_ai.system) or OpenLLMetry's llm.request.type is set
9emailemail.system is set
noneotherAnything else

Kind matters. A database span with kind INTERNAL is not a query, and an HTTP span with kind INTERNAL is neither a request nor an external call. An empty string counts as not set.

Within a category, spans are grouped for lists and charts by a group key:

CategoryGroup key
httphttp.request.method plus http.route, or url.path with identifiers replaced by :id
querydb.query.text with literals, numbers and IDs replaced by ?, else the span name
externalserver.address, else the host of url.full
jobmessaging.job.name, then messaging.bullmq.job.name, then messaging.destination.name
cachecache.key with ID and email segments replaced by *
emailemail.template, else email.message.subject with numbers replaced by #
clicli.command.name, then process.command, then process.executable.name
mcpmcp.tool.name, mcp.prompt.name or mcp.resource.uri, else mcp.method.name
aigen_ai.response.model, then gen_ai.request.model, then gen_ai.tool.name

Any category falls back to the span name when its attributes are missing. Set http.route to a template such as /orders/:id rather than relying on path guessing.

Service segments

A segment is the part of a trace that ran inside one service. Its first span is the entry point, and it holds the route, job or command context that the segment's queries, AI calls and exceptions are shown with.

A span starts a segment when:

  • it has no parent, or
  • its kind is SERVER, CONSUMER or INTERNAL and the OTLP span flags say the parent is remote.

Apart from root spans, CLIENT and PRODUCER spans never start a segment. When the span flags do not say whether the parent is remote, SERVER and CONSUMER spans start a segment and others do not. A root CLIENT span from a backend service is flagged as a background span.

Legacy attribute names

Ingest renames older attribute names to the current ones. If a span carries both, the current name wins.

Old nameCurrent name
http.methodhttp.request.method
http.status_codehttp.response.status_code
http.targeturl.path
http.urlurl.full
net.peer.nameserver.address
net.peer.portserver.port
db.systemdb.system.name
db.statementdb.query.text
db.operationdb.operation.name
db.sql.tabledb.collection.name
db.namedb.namespace
messaging.operationmessaging.operation.type
messaging.destinationmessaging.destination.name
enduser.iduser.id
enduser.roleuser.roles

GenAI names from older semconv releases, OpenLLMetry, Mastra and the Vercel AI SDK are renamed the same way, before token counts and cost are read:

Old nameCurrent name
gen_ai.systemgen_ai.provider.name
gen_ai.usage.prompt_tokensgen_ai.usage.input_tokens
gen_ai.usage.completion_tokensgen_ai.usage.output_tokens
gen_ai.usage.input_tokens.cached, gen_ai.usage.cache_read_input_tokens, gen_ai.usage.cached_input_tokensgen_ai.usage.cache_read.input_tokens
gen_ai.usage.input_tokens.cache_write, gen_ai.usage.cache_creation_input_tokens, gen_ai.usage.cache_creation.input_tokensgen_ai.usage.cache_write.input_tokens
gen_ai.usage.output_tokens.reasoning, gen_ai.usage.reasoning_tokens, llm.usage.reasoning_tokensgen_ai.usage.reasoning.output_tokens
gen_ai.tool.inputgen_ai.tool.call.arguments
gen_ai.tool.outputgen_ai.tool.call.result
ai.usage.costgen_ai.usage.cost

Resource attributes

AttributeUse
service.nameOverwritten with the name of the service your DSN belongs to
service.versionThe release. A new version on entry spans records a deployment
vcs.ref.head.revisionFull 40-character commit SHA, linked to the deployment
vcs.repository.url.fullRepository URL, stored with the resource
deployment.idYour platform's deployment ID, stored with the resource
service.instance.idIdentifies the process for system metrics; host.name is the fallback
host.nameHost name
deployment.environment.nameOverwritten with the x-unhandled-env header
telemetry.sdk.languagewebjs marks a browser service

service.version values 0.0.0, 0.0.1, 1.0.0, unknown, dev and latest are treated as missing. Unhandled's agents also set unhandled.service.version.source (where the version came from), unhandled.language.name, unhandled.framework.name, unhandled.sdk.name and unhandled.sdk.version.

HTTP requests

SERVER spans.

AttributeUse
http.request.methodRequired for the http category
http.routeRoute template used for grouping
url.pathGrouping fallback when there is no route
http.response.status_code4xx and 5xx counts and error rates
adonis.route.nameNamed AdonisJS route. Stored, not read by any view
unhandled.http.request.body, unhandled.http.response.bodyCaptured bodies, shown on the span

Database queries

CLIENT spans.

AttributeUse
db.system.nameRequired. redis is excluded from queries
db.query.textStatement, normalized for grouping. Send parameterized SQL
db.operation.nameOperation such as SELECT
db.collection.nameTable
db.namespaceDatabase name
db.transactiontrue for statements inside a transaction

External calls

CLIENT spans.

AttributeUse
http.request.methodRequired for the external category
server.addressHost, used for grouping
url.fullFull URL; the host is the grouping fallback, the path the pattern fallback
url.templateURL pattern per host, such as /orders/{id}
http.response.status_codeA status of 400 or more counts as an error

Exceptions

Three shapes become exceptions:

  1. Span events named exception, the OpenTelemetry default from span.recordException(). Unhandled reads exception.type, exception.message and exception.stacktrace, treats exception.escaped=true as unhandled, and copies user.id, user.email and user.name from the span.
  2. Log records with unhandled.log.source=exception_reporter, which is how Unhandled's SDKs report errors. Besides the three exception.* attributes they carry:
AttributeUse
unhandled.exception.valuesJSON array of the error and its causes, with mechanism and frames
unhandled.exception.values_versionFormat version of the field above, currently 1
unhandled.exception.framesJSON stack frames of the root error
unhandled.exception.cause_chainJSON list of causes
unhandled.exception.occurrence_idUnique ID that deduplicates retries
unhandled.exception.should_reportfalse keeps the occurrence out of issues
unhandled.message.levelLevel for captured messages, such as warning
unhandled.tag.*Tags
unhandled.extra.*Extra context
  1. Any other log record with exception.type or exception.message, as standard logging bridges send for a logged error. It becomes an occurrence with mechanism log_record and handled: true, with its level taken from the record's severity. Records that carry any unhandled.log.source value skip this path.

exception.stacktrace is parsed into frames for Python tracebacks, Go stack dumps, Ruby backtraces, Java stack traces and V8 stacks. Library frames are marked as not in-app: node_modules and node: for JavaScript, site-packages, dist-packages and the standard library for Python, pkg/mod, vendor and the Go runtime for Go, gems and the standard library for Ruby, and java.*, javax.*, jdk.*, sun.*, kotlin.* and scala.* for Java.

Exceptions are grouped into issues by a fingerprint of the type and the top 3 in-app stack frames, or the type and the normalized message when no frame is in-app.

Logs

Every log record is stored with its body, trace and span IDs and attributes. The level comes from severityNumber (1 to 4 trace, 5 to 8 debug, 9 to 12 info, 13 to 16 warn, 17 to 20 error, 21 and above fatal). Without a number, severityText is read case-insensitively: information maps to info, warning to warn, critical to fatal. Anything else is info. Records under the scope unhandled.analytics are analytics events, not logs. unhandled.log.source=console marks logs captured from the browser console.

Users

AttributeUse
user.idRequired to count users
user.emailShown with the user
user.nameShown with the user

Set them on the SERVER span of the request. enduser.id is renamed to user.id at ingest.

Jobs

AttributeUse
messaging.systemRequired. Unhandled's instrumentations send bullmq or boringqueue
messaging.operation.typesend or create for a dispatch, process for an execution
messaging.job.nameJob name, used for grouping
messaging.destination.nameQueue
messaging.message.idJob ID
messaging.message.retry.countAttempts so far
messaging.job.queue_time_msTime spent waiting in the queue
messaging.batch.message_countJobs in a bulk dispatch

A PRODUCER span, or any span with messaging.operation.type of send, publish or create, counts as a dispatch. Link or parent the execution span to the dispatch span so the two show up together.

Cache

AttributeUse
cache.operationRequired, for example get, set or getOrSet
cache.keyKey, sanitized for grouping
cache.hittrue or false, used for the hit rate
cache.storeStore name

Email

Any span with email.system counts as an email. No OpenTelemetry convention exists for email, so any SDK can set these attributes. The AdonisJS agent's mail instrumentation sets them on CLIENT spans.

AttributeUse
email.systemRequired. The agent sends adonisjs-mail
email.operationsend
email.transportMail transport
email.message.subjectSubject, used for grouping
email.message.from, email.message.toSender and recipients (JSON array)
email.message.cc_count, email.message.bcc_countCopy counts
email.message.has_attachmentsAttachments flag
email.message.idMessage ID
email.recipient_countRecipient count

Commands

A span is a command when one of these holds:

  • entry_point.type is cli, which the AdonisJS agent sets.
  • unhandled.entry_point is cli. Any SDK can set it on the root span of a command.
  • The span kind is INTERNAL and the span carries process.executable.name, as the OpenTelemetry CLI span convention describes. The attribute must be on the span. A resource attribute does not count.
AttributeUse
entry_point.type, unhandled.entry_pointcli marks a command
cli.command.nameCommand name, grouping
process.command, process.executable.nameName fallbacks, in this order
cli.command.descriptionDescription
cli.args.*Arguments
cli.flags.*Flags

AI

AttributeUse
gen_ai.operation.nameFor example chat, execute_tool or invoke_agent; decides how usage is counted (below)
gen_ai.request.model, gen_ai.response.modelModel, used for grouping and cost
gen_ai.provider.nameProvider
gen_ai.usage.input_tokens, gen_ai.usage.output_tokensToken counts. Input includes cached tokens, as semconv defines it
gen_ai.usage.cache_read.input_tokensInput tokens read from the provider's cache, priced at the cache-read rate
gen_ai.usage.cache_write.input_tokensInput tokens written to the cache, priced at the cache-write rate
gen_ai.usage.reasoning.output_tokensReasoning tokens, part of the output tokens
gen_ai.usage.costCost in USD reported by the provider or gateway; used instead of the estimate
gen_ai.tool.nameTool name
gen_ai.conversation.idGroups calls into a conversation
gen_ai.function_id, gen_ai.agent.nameAgent or function name
gen_ai.system_instructions, gen_ai.input.messagesPrompt content
gen_ai.output.messagesResponse content; its text parts are shown as the response of a conversation step
gen_ai.tool.call.arguments, gen_ai.tool.call.resultTool payloads

One count per model call. Agent and workflow spans (invoke_agent, create_agent, invoke_workflow) usually repeat the summed usage of the model calls inside them. Calls, tokens and cost count every other span with an operation (chat, generate_content, text_completion, embeddings and so on), except execute_tool and the retrieval and memory operations. An agent span counts only when no model call that reported usage started inside its time window. This covers frameworks that emit only the agent total, and remote agents called next to ordinary model calls. Only the outermost such agent span counts.

Cost. Unhandled prices usage when you look at it, from a dated table of list prices, so adding a model prices its earlier calls too and a price change only applies from its date. Bedrock (anthropic.claude-…), Vertex, OpenRouter (anthropic/claude-…) and dated model ids match the model they name. A call that sends gen_ai.usage.cost keeps that cost. A model without a price shows as unpriced rather than $0.

Content. Prompt, response and tool payload attributes are kept up to 256 KiB each, the same limit for every content key, including OpenLLMetry's gen_ai.prompt.* and gen_ai.completion.*. Before they are stored, they go through the same secret rules as exception data, plus an email rule, and each match becomes [Filtered]. The rules cover private keys, JWTs, bearer tokens, credentials in URLs, provider API keys, password= and "password": "..." style secrets, and card numbers that pass the Luhn check. Operators can turn rules off with INGEST_AI_SCRUB_DISABLE (comma-separated rule names: private_key, jwt, bearer, url_credentials, api_key, secret_assignment, secret_field, card, email) and add patterns with INGEST_AI_SCRUB_PATTERNS (a JSON array of regular expressions). Deleting a project erases its AI content right away, and node ace ai:erase-content --project <id> [--from <iso>] [--to <iso>] erases a range on demand.

Evaluations are span events named gen_ai.evaluation.result with gen_ai.evaluation.name, gen_ai.evaluation.score.value, gen_ai.evaluation.score.label and gen_ai.evaluation.explanation.

MCP

AttributeUse
mcp.method.nameRequired, for example tools/call
mcp.tool.nameTool
mcp.client.name, mcp.client.versionCalling client
mcp.tool.result.is_errortrue counts as a failure
rpc.jsonrpc.error_code, rpc.jsonrpc.error_messageJSON-RPC error, counts as a failure

Sampling

If you head-sample spans, set unhandled.sampling.ratio on each kept span to the ratio that kept it, from 0.000001 to 1. Unhandled counts such a span 1 / ratio times in call counts, error counts and average durations, so a service sampled at 0.25 shows its full traffic. A missing or out-of-range value counts the span once. @unhandled/node sets the attribute when samplingRatio is below 1.

System metrics

The system metrics view reads process.cpu.utilization, system.cpu.utilization, process.memory.usage, v8js.memory.heap.used, v8js.memory.heap.limit, nodejs.eventloop.utilization and nodejs.eventloop.delay.p99, per service.instance.id. Every other metric is stored and available in metrics.

Reserved Unhandled attributes

The server sets unhandled.organization.id, unhandled.project.id, unhandled.environment and unhandled.service.id from the authenticated request. Values you send under these keys are overwritten.

Attributes starting with unhandled. are hidden in the UI, except unhandled.http.*, unhandled.tag.* and unhandled.extra.*.