Alerts
Endpoint alerts on latency, error rate and traffic, with evaluation timing, the alert state machine, previews, limits and notifications.
An alert watches one HTTP endpoint of one service in one environment, and notifies you when a metric crosses a threshold. Put alerts on the routes that cost money when they break, such as POST /checkout or GET /api/orders/:id, so you hear about a slow or failing response before a customer reports it.
Open Alerts in the project sidebar to see the alerts of the current environment. The page has three tabs: Endpoint for the alerts on this page, Exceptions for exception rules, and Uptime for monitors. On Endpoint, filter by All, Firing, OK or Paused, and search by name or with service: and state:, as in service:api state:firing. Every member with access to the project can see alerts and their history, snooze them, and respond to incidents. Creating, editing, pausing and deleting alerts needs an admin.
Pick the endpoint
Start from Create alert on an endpoint in Requests, and the form arrives with the environment, service, method and route filled in. Or click New alert on the Alerts page and choose them yourself. The form suggests the busiest endpoints of the environment.
Set the condition
Choose the metric, the comparison, the threshold and the window. The condition section explains each setting.
Check the preview
While you edit, the preview replays the condition against recent traffic and marks when the alert would have fired. Adjust the threshold until it fires on the incidents you care about and stays quiet otherwise.
Choose destinations
Pick at least one notification channel, or turn on App members. See Name and destinations.
Save
Click Create alert. The alert starts in OK and Unhandled evaluates it on the next minute.
Condition
An alert reads as a sentence: metric of method route in service is above or below threshold over the last window.
Metricchoicedefaultp95 latencyp50 latency, p95 latency, p99 latency, Average latency, Error rate, 5xx error rate or Request rate. Error rate counts 4xx and 5xx responses. 5xx error rate counts only 5xx.
ComparisonchoicerequiredAbove or below. The comparison is strict, so a value equal to the threshold does not breach.
ThresholdnumberrequiredAny number from 0 up. Milliseconds for latency, percent for the error rates, requests per minute for Request rate.
Windowchoicedefault5 min5, 10, 15 or 30 minutes.
Must breach forchoicedefault2 minFire immediately, 2, 5 or 10 minutes.
Minimum requestsnumberdefault201 to 100,000. A window with fewer requests is ignored, so a quiet endpoint does not fire on a handful of calls. Request rate alerts skip the minimum, since low traffic is what they measure.
Alert if no trafficbooleandefaultoffTreat a window under the minimum as a breach instead of ignoring it. Turn it on to catch an endpoint that stopped receiving traffic.
Name and destinations
Leave Name empty and Unhandled builds one from the condition, for example p95 latency · POST /checkout > 800ms. A name can be up to 200 characters.
An alert needs at least one destination. Pick up to 20 of the organization's notification channels, and turn on App members to also notify everyone with access to the project. Admins can add a channel without leaving the form.
Evaluation
Every minute Unhandled evaluates each active alert over a window that ends 2 minutes before the current minute. The delay leaves room for spans that arrive late, so no window is judged on partial data.
If the background worker was down, it evaluates the minutes it missed when it comes back, oldest first, up to 5 of them. Older minutes are logged as skipped. The alerts list warns when evaluation has not finished for 3 minutes.
States
| State | Meaning |
|---|---|
| OK | The last window did not breach. |
| Pending | The window breaches, but not for long enough yet. |
| Firing | The window has breached for the Must breach for duration. An incident is open. |
| Recovering | The window stopped breaching, but not for the Resolve after duration yet. The incident stays open. |
| Paused | Unhandled does not evaluate the alert. |
| Error | The last 3 evaluations failed. |
An alert moves between states like this:
- OK becomes Pending on the first breaching window. With Fire immediately, OK goes straight to Firing.
- Pending becomes Firing once the breach has lasted the Must breach for duration. Only evaluated minutes count, so minutes the worker missed never make an alert fire sooner.
- Pending returns to OK when a window stops breaching. Nothing fired, so nothing is sent.
- Firing becomes Recovering when a window stops breaching, or when it falls under the minimum with Alert if no traffic off. A breaching window sends it back to Firing without a new incident or notification.
- Recovering returns to OK once the condition has stayed clear for Resolve after (5 minutes by default). The incident resolves. With Resolve immediately, Firing goes straight to OK.
- Any state becomes Error after 3 failed evaluations in a row, and the alert's destinations get one
alert.errornotification. The next successful evaluation returns the alert to Firing if its incident is still open, and to OK otherwise.
While an alert fires, Unhandled records the worst value it sees as the incident's peak. Each incident sends one firing and one resolved notification. With Remind while firing set to every hour, every 4 hours or every day, it also sends an alert.reminder at that interval while the incident stays open.
Snooze and acknowledge
Both keep an alert evaluating and its incidents opening and resolving. They only hold back notifications.
Snooze on the alert page, or in the row menu on the list, holds notifications for the whole alert: For 1 hour, For 4 hours, Until tomorrow 9:00 in your time zone, or Until resolved, which is offered only while the alert fires and lasts until that incident resolves. The alert shows a Snoozed chip with the time left. While it is snoozed:
- Reminders are skipped.
- A firing notification waits until a timed snooze ends, then goes out if the incident is still open. With Until resolved it is not sent.
- If an incident that was never announced resolves, its resolved notification is not sent either.
Unsnooze ends the snooze at once. Anyone who can see the project can snooze and unsnooze; the alert records who snoozed it.
On the alert page, each open incident has Acknowledge, an assignee and comments. Anyone who can see the project can use them. Acknowledging records who took it and stops reminders for that incident. The incident still resolves and sends its resolved notification as usual. An acknowledgement cannot be undone. Uptime monitor incidents have the same controls on the monitor page.
Pause, edit and delete
- Pause stops evaluation. A firing alert resolves first, with the reason Paused, and sends its resolved notification. Resume puts the alert back in OK and evaluation starts again.
- Editing the condition resets the alert to OK, and an open incident resolves with the reason Rule edited. The condition is the service, method, route, metric, comparison, threshold, window, must breach for, minimum requests and the no-traffic setting.
- Editing the name, the channels or App members keeps the current state.
- Changing the service, method, route or metric starts a fresh chart history. Past incidents stay.
- Delete, in the ⋯ menu, removes the alert and its incidents. A firing alert first sends its resolved notification, with the reason Deleted.
Preview
The preview replays the condition against recent traffic in 5 minute steps and shows when the alert would have fired. It covers the last 24 hours by default, or the last hour. p50 and p99 previews always cover the last hour.
For latency metrics the preview also suggests a threshold: the metric measured over the whole preview range, times 1.5.
Alert history
The alert page charts the evaluated values with deployment markers. Below the chart it lists the incidents with their start, end, peak and resolve reason, and the last notification sent. The resolve reasons are Recovered, No data, Rule edited, Paused and Deleted. Unhandled keeps evaluations for 30 days.
Notifications
When an alert fires or resolves:
- Each selected channel gets a message, retried up to 6 times over about 50 minutes if the channel fails. Webhook channels receive
alert.firing,alert.reminder,alert.resolvedandalert.errorevents, each with anincidentIdand adedupKeythat stays the same from firing to resolved. Notifications documents the payloads. - Slack and Discord channels wait 20 seconds before posting, so alerts that fire together arrive as one message that lists them. The same goes for member emails.
- With App members on, everyone with project access gets an in-app notification. Members whose Endpoint alerts email preference is on also get an email with the subject
[Alert] <name>or[Resolved] <name>.
Limits
| Limit | Value |
|---|---|
| Alerts per environment in a project | 50 |
| Active alerts per organization | 50 |
| Channels per alert | 20 |
Paused alerts do not count as active. Resuming one counts against the active limit again.
Alerts and the MCP server
The MCP server can create and change alerts under the same rules. An alert created there starts paused unless the tool call enables it. An alert without a service shows Needs service and Unhandled does not evaluate it until you pick one.
FAQ
The window breaches, but not yet for the Must breach for duration. It fires once the breach lasts that long, and returns to OK as soon as one window stops breaching.
The window had fewer requests than Minimum requests, so Unhandled ignored it. Lower the minimum, or turn on Alert if no traffic if silence is itself the problem.
When no service of the project delivers spans in the environment for a minute, Unhandled skips that minute. The alert keeps its state until spans arrive again.