Every software organisation learns most about its software on the day it fails. This chapter treats what a development organisation does when a change that AI helped to write, review or ship has broken something: how the incident is run, how the post-incident review is held without blame and without losing the name of the owner, how failures in which AI took part are analysed by mechanism, what verified public incidents show, and what is said to whom. It concerns D4 Prove, because an incident is the late and expensive verdict on whether the proof was sufficient, and D5 Steward, the function of owning software over time, which includes the authority to revert. The running of the incident itself, the commander, the runbooks and the restoration of service, belongs to O2 Diagnose and O4 Restore in AI Operations, and this chapter stops at that boundary.
The chapter makes one claim. An incident in which AI took part is analysed by the same method as any other, with a short list of additional questions: who owned the change, what the tool did, what the oracle was, and who could stop it. The questions do not move blame to the tool or to the person nearest to it. They find the condition in the system that let the change reach production without anyone being able to explain or undo it, using the stewardship principle of Chapter 4: Name, Understand and Revert.
Four kinds of event reach a development organisation under the heading "AI incident", and they need different handling. The first is a defect in generated or AI-assisted code that passed review and tests and failed in production. The second is an action by an agent that exceeded what its owner intended. The third is a toolchain incident, in which the coding tool, its extension or its build chain fails or is compromised. The fourth is a provider-side change, in which the behaviour or quality of a model changes without any change in the organisation's repositories. The most frequent event of all is the ordinary failure of ordinary software, in which AI took no part in the cause but may take part in the response, and its review is subject to the same rules.
Primary topics explored
- What breaks, and what is different when AI took part
- Post-incident practice with AI-involved change
- Blameless and named
- Analysing AI-involved failures
- Verified public incidents
- Communication
About AI Developer
AI Developer is Book 12 of The Transformed Architecture. It takes the developer as its subject and asks what happens when AI is used broadly in software development: the five developer functions (frame, design, build, prove, steward), the delegation ladder and the Level 4 prohibitions, the stewardship principle (name, understand, revert), verification as the scarce skill, the junior pipeline, adoption and productivity as claims that need evidence, regulation (the AI Act, the Cyber Resilience Act, the Product Liability Directive, NIS 2, DORA, the GDPR), ethics, intellectual property and sovereignty in the toolchain, and the developer as a person, for European enterprises and public authorities.
It builds on the conclusions of Architecture as Code, AI Governance, AI Platform and Security, AI Red Team, AI Threat, The Transformed Security Architect, AI Boss, AI Blue, AI Purple, AI Operations and AI Leader.
developers at every level, tech leads and engineering managers, architects, platform engineers, site reliability engineers and testers, product owners and designers, leaders who decide on developer tooling, heads of public-sector IT and procurement, works councils and unions, educators, and supervisors and auditors.
Gunnar Nordqvist
Conceived, directed and published the books of The Transformed Architecture.
Continue with the full chapter
Chapter 23: When It Breaks: Incidents, Outages and AI-Generated Defects in AI Developer gives the detailed explanations, practical examples and implementation patterns behind this summary.
This whitepaper belongs to “Leading Change and Adoption” and is one of 37 drawn from the manuscript.