Job sampling

Decide per job which background job traces to keep, drop or rate-limit, and keep failed runs even when you sample successes away.

A heartbeat job that runs every few seconds, or a newsletter fan-out that sends thousands of emails, can produce more traces than the rest of your app combined. Job sampling lets you decide, job by job, which executions to keep. You can drop a noisy job entirely, keep a fraction of it, or drop every successful run and keep every failure. It is configured in the AdonisJS agent with a sampling callback in config/unhandled.ts.

Configure

import env from '#start/env'
import { defineConfig, Sample } from '@unhandled/agent-adonisjs'

export default defineConfig({
  dsn: env.get('UNHANDLED_DSN'),
  environment: env.get('APP_ENV'),
  sampling: (job) => {
    if (job.name === 'ProcessOrder') return Sample.always()
    if (job.name === 'PingWarehouse') return Sample.never()
    if (job.name === 'SyncProductFeed') return Sample.rate(0.1)
    if (job.name === 'SendNewsletterIssue') return Sample.rate(0).keepErrors({ maxSpans: 500 })
    return Sample.default()
  },
})

The agent calls the callback once per job execution and applies the rule it returns. Return Sample.default() for everything you do not want to change.

Job context

The callback receives one argument describing the execution:

Field	Type	Content
`kind`	`'job'`	Always `'job'`.
`system`	`string`	The queue system: `boringqueue` for `@adonisjs/queue`, `bullmq` for BullMQ.
`name`	`string`	The job name.
`queue`	`string` or undefined	The queue name, when the system reports it.
`jobId`	`string` or undefined	The job id, when available.
`attempt`	`number` or undefined	The attempt number the queue system reports, when it reports one. BullMQ counts the first attempt as `1`.

The types are exported as JobSamplingContext and SamplingCallback from @unhandled/agent-adonisjs and @unhandled/agent-adonisjs/types.

import { defineConfig, Sample } from '@unhandled/agent-adonisjs'

export default defineConfig({
  sampling: (job) => {
    if (job.queue === 'payments') return Sample.always()
    if (job.name.includes('Import')) return Sample.rate(0.01).keepErrors()
    return Sample.default()
  },
})

Rules

Rule	Effect
`Sample.default()`	No change. The execution follows the normal head sampling.
`Sample.always()`	Every execution is exported.
`Sample.never()`	No execution is exported.
`Sample.rate(r)`	Each execution is kept with probability `r`, from `0` to `1`. A value outside that range throws a `RangeError`.
`.keepErrors(options?)`	Chained on any rule. Executions the rule would drop are buffered and exported anyway if they fail. It has no effect on `Sample.default()`, which never drops.

Sample.rate(0).keepErrors() is the rule to reach for when only failures matter: every successful run disappears and every failed run is kept in full.

Without keepErrors(), a dropped execution is dropped even if it fails.

How keepErrors works

keepErrors() decides after the job finishes, inside your process:

When the job's root span starts, the rule is evaluated. If the execution is sampled in, its spans are exported normally.
Otherwise the job's spans are held in memory as they end.
When the root span ends, the agent checks whether the root or any buffered span ended with ERROR status. If so, the root and then the buffer are exported. If not, the buffer is discarded.

keepErrors({ maxSpans }) caps the buffer. A job keeps its first maxSpans spans, and later spans are only counted. If that job then fails, the root is exported first, followed by the kept spans, and the root carries two attributes:

Attribute	Value
`unhandled.sampling.truncated`	`true`
`unhandled.sampling.dropped_spans`	Number of spans that were not kept.

Without maxSpans, the cap is one less than batch.maxQueueSize (2,047 by default), so a failed job and its root fit in an empty export queue. Each running job holds at most that many spans in memory.

Because the decision comes after the fact, every trace must be recorded first. Setting sampling therefore forces samplingRatio to 1. Spans that never get recorded cannot be recovered by keepErrors().

Queue systems

@adonisjs/queue

Executions are recognized from the job's CONSUMER span, which carries messaging.system. The rule is evaluated when that span starts and applies to every span below it. All rules work, including keepErrors(). Spans of a dropped execution are still created in memory but never exported. Other instrumentations that create CONSUMER spans with messaging.system are sampled the same way.

BullMQ

With @unhandled/instrumentation-bullmq registered in integrations, the rule is evaluated before the worker runs the job. When the execution is dropped, tracing is suppressed for the whole run, so it creates no spans at all.

BullMQ uses the agent's sampling callback by default. Pass sampling to bullmq() to use a different callback for BullMQ jobs only:

import { defineConfig, Sample } from '@unhandled/agent-adonisjs'
import { bullmq } from '@unhandled/instrumentation-bullmq/integration'

export default defineConfig({
  sampling: (job) => (job.name === 'PingWarehouse' ? Sample.never() : Sample.default()),
  integrations: [
    bullmq({
      sampling: (job) => (job.queue === 'emails' ? Sample.rate(0.05).keepErrors() : Sample.default()),
    }),
  ],
})

See BullMQ.

What sampling does not touch

Logs, metrics and exceptions you report with Unhandled.captureException are sent as usual. Only spans are filtered.
Requests, Ace commands and other non-job work are not passed to the callback.
Views built from spans, such as Jobs and Traces, only see exported executions, so a sampled job shows lower throughput than it really has. samplingRatio is different: it stamps unhandled.sampling.ratio on every span it keeps, and the per-minute rollups count each kept span as 1/ratio calls. Job rules stamp no ratio, so their counts are not scaled back up.

Job sampling is available in the AdonisJS agent only. See Configuration for samplingRatio, which samples every trace by a fixed ratio.