This chapter concerns O4 Restore, the function that returns a service to an acceptable state: remediation, rollback, failover and recovery from backup. It touches O2 Diagnose at its start, because an action that destroys the evidence of a fault has destroyed the means of diagnosing it, and O3 Change at its edge, because most remediation is a change made under pressure. The chapter's debate is whether AI may remediate production on its own. Restarting a failed process, adding an instance and rolling back a release are old candidates for automation, and they are also the actions by which an operator's agent holds the broadest credentials in the estate. The chapter answers per action class: it sets out the runbook as code, four remediation tiers, the actions an agent may and may not take, safe execution, independent verification, and the backup and restore that remain when everything else has failed.
Three rules from Chapter 4 govern everything below. The degree of delegation is set per action class and bounded by what can be undone, how far a mistake spreads, the agent's evidenced record and whether the action falls inside an authorised window. A stop path runs outside the system it stops. What verifies an action is independent of what performed it (the observer principle, Chapter 17). The chapter adds a conclusion of its own: no remediation may run that the organisation could neither undo nor restore from.
O4 Restore has five ways of returning a service to an acceptable state, and they differ in what they risk. The first is to remediate in place: restart a process, scale out, clear a regenerable cache, renew a certificate. The second is to roll back what was deployed or configured. The third is to fail over to redundant capacity. The fourth is to restore data from a backup, beside the damaged copy or over it. The fifth is to repair forward, which is a change with its own review (Chapters 8 and 9). In the action classes of Chapter 4 these are restart, scale, reconfigure, roll back, fail over, delete and restore-overwrite, with read, annotate and notify as the supporting classes that surround every action.
Primary topics explored
- What restoring covers
- The runbook as code
- Remediation tiers
- The first ten actions
- Safe execution
- State engines as the runners of runbooks
About AI Operations
AI Operations is Book 10 of The Transformed Architecture. It takes IT operations as its subject and asks what happens when AI takes part in it: the five operations functions (observe, diagnose, change, restore, sustain), the autonomy budget and the Level 4 prohibitions, the observer principle, resilience as a claim that needs evidence, operating AI systems, the operator's own AI as Tier 0 and how to defend it, operating models, decision rights, regulation (DORA, NIS 2, the AI Act), and the people involved, for European enterprises and public authorities.
It builds on the conclusions of Architecture as Code, AI Governance, AI Platform and Security, AI Red Team, AI Threat, The Transformed Security Architect, AI Boss, AI Blue and AI Purple.
CIOs and heads of IT operations, SRE and platform leads, incident commanders and on-call engineers, service-desk and service-management leads, FinOps and capacity managers, architects, CISOs and heads of resilience and third-party risk, boards and management bodies, procurement and vendor managers, supervisors and auditors, works councils and unions, data protection officers, and public-sector leaders.
Gunnar Nordqvist
Conceived, directed and published the books of The Transformed Architecture.
Continue with the full chapter
Chapter 10: Restore: Runbooks, Auto-Remediation and Recovery in AI Operations gives the detailed explanations, practical examples and implementation patterns behind this summary.
This whitepaper belongs to “The Craft” and is one of 35 drawn from the manuscript.