Queries

Queries groups every database statement your services run into patterns, so you can see which SQL costs the most time in total, which is slowest at P95, and which runs too often inside one request. It is for anyone chasing a slow page down to the database.

The list lives at /o/:org/projects/:project/queries and a pattern at /o/:org/projects/:project/queries/detail?key=....

What counts as a query

A query is a CLIENT span with db.system.name set, except Redis: spans with db.system.name=redis are not queries. Cache commands belong on Cache when the span carries cache.operation.

ClickHouse calls show up with db.system.name=clickhouse when the service runs @unhandled/node or one of the framework packages built on it, which trace @clickhouse/client without setup (see ClickHouse). Without that instrumentation, a ClickHouse call is a plain HTTP request and appears on External calls instead.

The Drizzle and Kysely instrumentations set db.system.name, db.query.text, db.operation.name and db.collection.name. Older instrumentations that send db.system, db.statement, db.operation or db.sql.table work too, because ingest renames them to the stable names. See Semantic conventions.

How patterns are built

The pattern key is db.query.text after normalization, so the same statement with different parameters lands on one row:

• comments are removed and whitespace is collapsed
• string literals, numbers, $1 style placeholders, UUIDs, IPv4 addresses and hex strings of 12 or more characters become ?
• IN (?, ?, ?) and VALUES (?, ?) lists collapse to IN (?) and VALUES (?)
• keys longer than 1,000 characters are truncated

ClickHouse {name:Type} query parameters such as {day:Date} are kept as written, so a parameterized ClickHouse query is already one pattern.

select * from "orders" where "customer_id" = 42 and "status" in ('paid', 'shipped')
select * from "orders" where "customer_id" = ? and "status" in (?)

A span without db.query.text is keyed by its name. Patterns are per service: the same SQL from the API and the worker is two rows.

The list

Queries Over Time stacks successful and failed executions per bucket. Duration Over Time plots the median and P95, and the legend adds P99.

The table lists up to 200 patterns with Service, Op, Database, Query, Executions, Avg, P95, Total time and Errors. Op is db.operation.name, or the first word of the statement when it is missing. Database is db.system.name. The Service column is hidden when exactly one service is selected.

Sort defaults to Total, the sum of all execution time, because a 5 ms query that runs 100,000 times costs more than a 2 s report that runs twice. P95, Executions, Avg and Errors are the other orders. The Database filter lists every db.system.name seen in the range and limits the list to one of them.

Pattern detail

The detail page shows the full normalized SQL, then the cards and charts for that pattern alone. When more than one service runs the pattern and you arrived without a service, it asks you to choose one first.

Possible N+1 query appears when the pattern ran 5 or more times inside a single trace. It reports how many traces did so, the highest repeat count and the average. A loop that loads one row per item is the usual cause.

Latency distribution is a heatmap of executions by start time and duration. Drag a box over cells, or click Slowest 10%, to compare the executions inside the box with the rest of the range. The comparison lists, per attribute such as service, version, instance, database and operation, the values whose share differs most between the box and the rest, with Filter to and Exclude buttons. Each side counts at most 200,000 executions, so shares on busy patterns are estimates.

Callers lists the routes, jobs, commands and MCP tools whose traces ran the pattern, with calls, P95 and each origin's share of the total time. It reads the latest 20,000 executions in the range.

Slowest executions lists the 50 longest executions with Time, Origin, Status, Duration and a trace link. Origin is the entry point of the trace: the route, job, command or MCP tool that ran the query. Status filters to All or Failed, and Show switches to Recent executions, the 50 latest.

Answering "why is the orders page slow?"

1. Keep the sort on Total and look for patterns touching orders.
2. Open one and check for the N+1 panel. If it shows up, the page loads related rows one at a time.
3. Read Callers for the route that runs it most, then open the slowest trace in Slowest executions to see the request that issued it.
4. Fix the loop with a single query or an eager load, deploy, and compare Total time over the next day.