SLOs

An SLO (service level objective) sets the share of requests that must be good over a rolling window, such as 99.9% of POST /checkout requests answering without a 5xx over the last 28 days. Unhandled measures it, shows how much of the error budget is left, and can page you when the budget is burning too fast.

Open SLOs in the project sidebar to see the SLOs of the current environment. The list lives at /o/:org/projects/:project/slos. Every member with access to the project can see SLOs. Creating, editing and deleting them needs an admin.

What an SLO measures

An SLO targets one service and either one of its HTTP endpoints (METHOD /route, grouped as on the Requests page) or all of its HTTP endpoints. Its indicator, the SLI, decides which requests are good:

The SLI is good requests divided by all requests in the window.

Objective and window

The error budget is the share of requests allowed to be bad: 100% minus the objective. A 99.9% objective has a budget of 0.1%.

Where the numbers come from

Unhandled counts requests from its per-minute and per-hour request rollups, never from raw spans.

A latency SLI reads the per-minute latency histogram, whose buckets double in width. Counts are exact when the threshold is a power of two in nanoseconds. Otherwise Unhandled splits the bucket that contains the threshold on a log scale, the way Prometheus' histogram_fraction does, so the SLI can be off by at most the requests in that one bucket.

The window cannot reach further back than your organization's data retention. Requests older than that are not counted.

Create an SLO

An environment can have up to 50 SLOs.

Objective suggestion

The form shows the SLI the target achieved over the last 28 days and suggests the highest of 99.99, 99.95, 99.9, 99.5, 99, 98, 97, 95 and 90 that is strictly below it, so a new SLO starts with budget to spare. Below 90%, it suggests the whole percent under the achieved value. Without traffic in those 28 days there is nothing to suggest. The suggestion is only a starting point; type any objective you like.

The list

Each SLO shows its name and target, its indicator, the current SLI, the objective, the error budget left, a burndown over the window, the window, and the state of its burn-rate alerts.

Budget left is how much of the window's budget remains:

budget left = 1 − (bad requests / all requests) / (1 − objective)

It starts at 100% and drops as bad requests arrive. It goes below 0% once the window has spent more than its whole budget.

The burndown is the budget left at the end of each UTC day of the window, measured against the whole window's budget: it starts at 100% and falls with each day's bad requests to where the window stands now. The first and last days are partial.

SLO detail

Click an SLO to open it. The page shows:

• SLI, Objective, Budget left and Good requests, with the total next to the good count.
• Burndown: the budget left at the end of each day of the window.
• Budget consumed by day: each day's bad requests as a share of the whole window's budget, so the days add up to what the window has consumed.
• Burn rate: two panels over the last 24 hours, one for the 1 hour and 5 minute windows, one for the 6 hour and 30 minute windows, each with its alert threshold.
• Burn-rate alerts: the alerts that watch this SLO and their state.

Burn rate

The burn rate is the error rate divided by the error budget. At 1x the budget runs out exactly at the end of the window. At 2x it runs out halfway through.

Burn-rate alerts

With Burn-rate alerts on, creating an SLO also creates two alerts, following the multi-window, multi-burn-rate alerts of the Google SRE workbook:

The long window shows the burn is sustained, and the short one that it is still happening. Both alerts fire immediately, without a Must breach for delay, and notify the destinations you picked on the form.

They are regular alerts with the subject SLO. They appear on the Alerts page, count toward the alert limits, and get the same evaluation every minute, states, snooze, incidents and notifications as any other alert. Edit, pause or delete them there. To add your own, pick SLO as the subject on the alert form, see Alerts.

Edit and delete

• Editing an SLO changes its target, indicator, objective, window or name. Its burn-rate alerts keep their settings and follow the SLO's new service.
• Delete, in the ⋯ menu, removes the SLO and its burn-rate alerts with their incidents. A firing alert first sends its resolved notification, with the reason Deleted.

SLOs and the MCP server

The MCP server can list, read, create, change and delete SLOs with list_slos, get_slo, create_slo, update_slo and delete_slo. create_slo adds the two burn-rate alerts only with burnAlerts: true, and takes the suggested objective when you leave objectivePercent out, or 99.9 when the target had no traffic. Burn-rate alerts created there start paused unless the call passes enabled: true.

FAQ

Related