Incidents

Nines opens incidents automatically when a monitor degrades. Two incident types map to two detectors.

Incident types

Region-failure incident
Opened when a strict majority of probe regions report down or error on the most recent check. Visible on the dashboard and on the monitor's public status page.
Burn-rate incident
Opened when SLO error budget is consumed faster than it refills. Two SLI types: availability_burn and latency_burn. Visible to the account owner only — never on a public status page. See Burn-rate incidents and Incident detectors.

Region-failure lifecycle

  1. Investigating — set on creation when failures are detected.
  2. Identified — set manually via the incident detail page.
  3. Monitoring — set manually via the incident detail page.
  4. Resolved — set automatically when all regions report up. Auto-resolve transitions directly to Resolved; Monitoring is operator-only and is skipped on the auto-resolve path.

Burn-rate incident lifecycle

Same four states as region-failure. Differences:

  • Auto-resolve only fires when burn rate is confirmed below threshold and stays below for a 5-minute cooldown.
  • If the metrics backend returns an error, returns no data, or the warmup gate is in effect, the detector returns Unknown. The incident stays open and the cooldown clock is paused.
  • Monitoring is operator-only; the auto-resolve path skips it and transitions directly to Resolved.

Notification events

incident.created
Webhook event and email dispatched when an incident opens.
incident.resolved
Webhook event and resolution email dispatched when an incident closes.
incident.updated
Webhook event dispatched when an incident's status or details change (e.g., transition to Identified).

Channel configuration: Notifications. Payload schema: Webhooks.

Viewing incidents

All open and recent incidents are listed on the Incidents page. Each row shows the monitor name, affected regions (region-failure only), open time, and status. The detail page shows the timeline and lets an operator update status or add a note. Burn-rate incidents display the SLI type and the burn multiplier at the time of detection.