Traces

Traces shows every unit of work your services recorded: requests, jobs, commands and background work. Use it when a dashboard tells you something is slow or failing and you need the one execution that explains why.

The list lives at /o/:org/projects/:project/traces and a single trace at /o/:org/projects/:project/traces/:traceId.

Where the data comes from

Every span your services export lands here: the AdonisJS agent for HTTP, queries, mail, queue and Ace commands, the instrumentations for libraries such as BullMQ and Bentocache, and any OpenTelemetry SDK. Spans join one trace only when trace context is propagated between services, for example through the traceparent header. Without it, each service produces its own trace.

The segment chart

The chart counts segments, not traces. A segment is the part of a trace that runs in one service: the root span, plus every SERVER or CONSUMER span whose parent lives in another process. A request from the web app to the API to a worker therefore counts three segments, one per service.

Chart mode switches between:

• Count: segments per bucket, with failures stacked on top.
• P95 duration and Avg duration of those segments.

Group splits the chart by any field. Common choices are offered first (service.name, http.route, status, span.kind, category, http.request.method), then attribute keys seen in recent spans. The 6 largest groups get their own series and the rest fold into Other. Drag across the chart to zoom the time range. The arrow next to Chart mode hides the chart, and the page remembers that choice.

Hide background spans removes traces whose root is a CLIENT span from a backend service: a query or HTTP call made outside any request, job or command. See Background spans.

The trace list

Each row is one trace, keyed by its root span. When the root span never arrived, for example because a load balancer, a gateway or a sampled-out parent started the trace, the row uses the earliest entry span of the trace instead and says missing root next to the name.

Status tabs filter on the root span status: All, OK, Error, Unset, each with its count. Click Time or Duration to sort newest or longest first. Columns adds any attribute of the root spans as a column. The menu on an attribute value filters in or out on it, groups the chart by it, copies it, or opens the same filter in Logs. For service.name it also opens Exceptions for that service. The density toggle switches to compact rows. Rows load 50 at a time as you scroll, and Up and Down move between rows. Copy link copies the current view, including search, columns and chart mode. Its menu also copies a link to the exact window, which pins the start and end so everyone sees the same data.

Newest-first pages read back from the end of the range in growing windows (15 minutes, 1 hour, 6 hours, 24 hours, then the rest) and stop once the page is full, so the first page of a 30-day range stays fast.

Top values, the rail on the left of wide screens, lists the most frequent values of common fields. Its counts cover the time range, not the search. Click a value to filter on it.

Live tail

While the list is sorted by Time, Live streams new traces into the top of the list. Starting it from a custom range switches the range to 15m. Polls run every 2 seconds and slow to every 5 seconds while nothing arrives. Each poll rereads the last 30 seconds so late spans still show up, and rows already shown are skipped. Live tail pauses while the tab is in the background and picks up again when you return. It also pauses when you scroll down the list, and Resume shows how many rows arrived meanwhile. When one poll holds more than 200 new rows, the newest are kept and the banner says some were skipped. Stopping keeps the window you watched as a custom range.

Search syntax

The search bar is shared with Logs. A query is a list of tokens separated by spaces, and every token must match.

• A bare word is free text. It matches span names case-insensitively, anywhere in the name.
• field<op>value is a filter. Fields start with a letter or _ and may contain letters, digits, _, . and -.
• Double quotes keep spaces in a value: name="GET /orders". An unclosed quote is an error.

A comma list with = or != means is any of or is none of: http.request.method=POST,PUT. Quoting the value turns the comma back into a literal character.

Comparisons need a number. http.status_code>high is rejected with "http.status_code needs a number to compare", and duration>slow with "Invalid duration". duration takes a unit: us, µs, ms, s, m or h, and a bare number means milliseconds, so duration>=1.5s and duration>=1500 are the same filter.

These span fields map to stored columns:

Any other field reads the span attribute of that name as text, and a numeric comparison on it converts the text to a number first.

A filter matches if any span in the trace matches, so db.system=postgresql keeps every trace that ran a Postgres query even though the root is an HTTP span. Pasting a 32-character lowercase hex trace ID opens that trace directly.

In the bar, typing a field name suggests fields, and typing an operator suggests the 25 most frequent values with their counts. Picking several values builds an is any of filter. Tab cycles the operator, Backspace on an empty input removes the last filter, and / focuses the bar from anywhere on the page. Clicking an operator on a filter chip cycles it too.

The trace view

The header shows the entry point, the trace ID with a copy button, span count, total duration, number of services, error count and log count. The log count links to Logs filtered to trace.id.

A trace with more than 5,000 spans loads 5,000 of them: every error span first, then the earliest and the latest. A banner says "Showing 5,000 of N spans". When the same span arrives twice, for example because an exporter retried a batch, it shows once.

Under the header, two summaries help you find the slow part before you read the waterfall:

• Slowest spans by self time and errors marks the 5 child spans with the most self time, and every failed span, on the trace's timeline. Click a mark to select that span.
• Time by service (self time) splits the trace's self time by service.

Self time is a span's duration minus the time its children cover. Children that run in parallel are subtracted once.

The waterfall nests spans under their parents, with a Self column next to each span's duration. Failed spans show in red with a warning icon, and log records appear as dots on their span's bar. A span whose parent never arrived, or whose parents form a loop, shows as a root. Above the waterfall:

• Find a span highlights spans whose name matches and keeps their ancestors visible. / focuses it.
• Focus a service and Errors only hide everything else, again keeping ancestors.
• Up and Down move the selection. Left collapses the selected span, or moves to its parent when it is already collapsed. Right expands. Escape closes the panel.

Selecting a span opens a resizable panel. Tabs only appear when the span has data for them: Overview, Attributes, Resource, Exception, Events, Logs, Links, Infra and JSON. Overview shows start and end time, the status message of a failed span, the SQL of a query with keywords and parameters highlighted, and the span, parent and trace identifiers. The panel shows the span at once and loads its full attributes when you open it.

A percentile chip such as p97 for GET /orders/:id ranks the span's duration among calls to the same operation in the hour before it. Hover it for that hour's P50, P95 and P99. The chip needs at least 20 calls to rank against.

Infra appears when the span's resource carries host.name or k8s.pod.name. It charts process CPU, process memory and the P99 event loop delay of the instance that ran the span, five minutes either side, with the span shaded.

Trace logs below the waterfall lists up to 1,000 log records of the trace, from 5 minutes before its first span to 5 minutes after its last. Clicking one selects its span.

Cross-service spans

When a span's parent is in another service, its row shows a service badge and the panel shows Called by, or Sent by for jobs, with the caller's name and service. For HTTP calls the grey bars on either side of the callee are network time. Jobs show Queue wait from messaging.job.queue_time_ms.

An outgoing CLIENT span and the SERVER span it reached in the same service merge into one row, so a service calling itself does not add a level.

Server clocks drift. When a callee appears to start before the outgoing call, or to end after the caller got its response, Unhandled moves it inside the call: centred when it is shorter than the call, aligned to the call start when it is longer. A span from another service that starts before a parent which is not an outgoing call moves to the parent start. Shifts of 1 ms or less are ignored and durations never change. A shifted span shows a clock icon, and its panel shows the original Recorded start, the shift and the reason. The header duration uses the corrected times.

Service map

Service Map, in the System group of the sidebar, draws the calls between your services from the same spans, at /o/:org/projects/:project/service-map. Callers sit on the left and dependencies on the right. Databases and external peers appear as their own nodes. A node's size follows its calls per minute, and its ring shows its error rate.

A connection exists when an entry span in one service has a parent span in another service in the same trace, so both services must propagate trace context. Click a node to isolate it. The side panel shows its rate, errors and P95 duration, what calls it and what it depends on, and links to Requests, Traces, Logs and Exceptions filtered to that service.

The map reads at most the last 7 days of the selected range. Above 2 million spans in that window it reads a fixed sample of traces and says so, and on a busy project it shows only the busiest 40 nodes and 120 connections.

Retention

Traces older than your plan's retention are hidden from every read, including the trace view and its logs. Storage drops expired data a whole day at a time, so it can stay on disk for up to a day after it stops showing.

Answering "why was this checkout slow?"

1. Search http.route=/checkout duration>=2s and sort by Duration.
2. Open the slowest trace and look for the widest bar under the root.
3. If it is a query, Overview shows the SQL. If it is a call to another service, the network bars tell you whether the time went to the wire or the callee.
4. Check Trace logs for warnings logged during the request.

Not available yet: a side preview of a trace from the list. A row opens the full trace view.