Observe with AI

Chapter 5 described how an operator sees a service without AI: signals, service level objectives, error budgets and the cost of keeping them. This chapter asks what changes when software that learns from telemetry is put between the signals and the person on call. It concerns O1 Observe, where the detection of operational anomalies and the shaping of alerts take place, and O2 Diagnose, where the first grouping of evidence and the first summary decide where an engineer starts looking. The chapter sets the gains of correlation, seasonality-aware baselines and language-model summaries beside their failure modes: the suppression of the one alert that matters, a baseline that drifts to the new broken normal, and dependence on the very data that fails. It then shows how to evaluate such a system with golden incident sets, precision and recall per alert class, a noise budget and seeded alerts, and what the published evidence does and does not establish.

The chapter builds on conclusions of the earlier volumes. A detection is a record with an owner, a test and an alert budget, and counts only when a recent test shows that it fires and a measure shows what it costs (AI Blue, Chapter 6, F95) (302, Nordqvist, 2026). Closure of alerts is delegated by class under a policy that states the evidence to close, the level, the sampling and seeding rates and a stop condition written as numbers, enforced outside the model (AI Blue, Chapter 7, F96). Observation is comparison with a written expectation sealed before the result is shown, with the ground truth held outside the system observed (AI Purple, Chapter 9, F125), and a check counts as verification only when its failure modes are not those of the thing it checks (AI Purple, Chapters 4 and 15, F118). Chapter 4 of this book states the same rule for operations as the observer principle. This chapter applies it to the first place where AI meets operations in most organisations, the alert stream.

A threshold alert compares one signal with one number. A burn-rate alert, the form Chapter 5 recommends for service level objectives, compares the rate at which an error budget is being spent with a rate that would exhaust it too early. The Site Reliability Workbook gives a worked set of burn-rate rules for a 99.9 per cent objective: a page when the budget is being spent fast enough to consume 2 per cent of it in one hour (burn rate 14.4, with a 5-minute short window), a page for 5 per cent in six hours (burn rate 6, with a 30-minute short window), and a ticket for 10 per cent in three days (burn rate 1, with a 6-hour short window) (244, Thurgood et al., 2018). The chapter reports good precision, good recall and good detection time for this multiwindow approach, and names its cost: many parameters to keep consistent. Rules of this kind are the anchor of the chapter, because they are written against what users experience. What AI adds is applied to the signals that surround them.

Primary topics explored

1. What AI adds to observing
2. Alert noise and its cost
3. Failure modes
4. Evaluation
5. What the evidence shows
6. A correlation policy