An operations organisation is steered by what it measures. A number on a dashboard decides where engineers spend the quarter, whether a release goes out on Friday, whether a recovery objective is believed and whether an agent keeps the permissions it was given. This chapter concerns all five operations functions defined in Chapter 4: O1 Observe, because every measure is computed from telemetry; O2 Diagnose, because incident durations describe how diagnosis went; O3 Change, because the best-known operations metrics describe the change pipeline; O4 Restore, because time to restore and the evidence for recovery measure it; and O5 Sustain, because toil, cost and the condition of the operators are measured over months and years. The chapter sets out which metric informs which decision, how each one misleads, and what a definition must contain so that the misuse is written down before it happens.
The argument has three parts. A metric is justified by the decision it informs. The usual operations measures fail in predictable ways: means hide tails, ratios hide missing data, counts reward volume, and any measure that becomes a target is optimised, which AI makes cheaper and faster. And the measures that matter for this book's thesis are new ones, such as the consumption of the autonomy budget, the share of actions verified by an observer independent of the actor and the age of the evidence for a recoverability claim; the chapter defines them with the same care as the standard ones.
The chapter builds on the earlier volumes. AI Blue, Chapter 20, replaced time to detect, time to respond and closure counts with four families of measure, each with a gaming check, and fixed the rule that measures about people are agreed first and reported in aggregate (F105) (302, Nordqvist, 2026). AI Purple, Chapter 18, described Goodhart's law in four forms and recorded that a green percentage without a denominator misleads (F132). AI Boss, Chapters 11 to 15 and 26, supplied the measurement of oversight quality, with the rule that an approval given in seconds is a signal of failure (F84), and the quiet harms of monitoring staff (F89) (301, Nordqvist, 2026).
Primary topics explored
- What a metric is for
- The four key metrics
- SLO attainment and error budgets
- Time to restore: the mean and the distribution
- Toil and the autonomy budget
- False greens and gaming
About AI Operations
AI Operations is Book 10 of The Transformed Architecture. It takes IT operations as its subject and asks what happens when AI takes part in it: the five operations functions (observe, diagnose, change, restore, sustain), the autonomy budget and the Level 4 prohibitions, the observer principle, resilience as a claim that needs evidence, operating AI systems, the operator's own AI as Tier 0 and how to defend it, operating models, decision rights, regulation (DORA, NIS 2, the AI Act), and the people involved, for European enterprises and public authorities.
It builds on the conclusions of Architecture as Code, AI Governance, AI Platform and Security, AI Red Team, AI Threat, The Transformed Security Architect, AI Boss, AI Blue and AI Purple.
CIOs and heads of IT operations, SRE and platform leads, incident commanders and on-call engineers, service-desk and service-management leads, FinOps and capacity managers, architects, CISOs and heads of resilience and third-party risk, boards and management bodies, procurement and vendor managers, supervisors and auditors, works councils and unions, data protection officers, and public-sector leaders.
Gunnar Nordqvist
Conceived, directed and published the books of The Transformed Architecture.
Continue with the full chapter
Chapter 20: Measuring Operations: The Four Key Metrics, SLOs, Toil and the Metrics That Mislead in AI Operations gives the detailed explanations, practical examples and implementation patterns behind this summary.
This whitepaper belongs to “Operations in the Organisation” and is one of 35 drawn from the manuscript.