Chapter 12 applied the loop to the organisation's business AI: the claims assistant, the benefits-triage service, the agents that act for staff. This chapter applies it to a different subject, the AI that sits on the blue team's side of the table: the agent that triages an alert, the agent that builds a timeline, the agent that drafts a detection, the playbook that isolates a workstation and the AI that a managed provider uses to close alerts at night. AI Blue argued that these systems are part of the estate that is defended and must be verified by review, seeded cases and a stop path (AI Blue, Chapters 14 and 15) (94, Nordqvist, 2026). It left a gap that this book can fill. A blue team can seed its own queue, but the realistic cases, the hostile inputs and the ground truth come most cheaply from the other half of the loop. The chapter concerns P2 Emulate, where emulation produces the seeded incident and the hostile input; P3 Observe, where the behaviour of the agent is compared with what the scenario implies it should have done; P4 Improve, where a failure becomes a change in a mandate, a playbook or a contract; and P5 Verify, where the change is re-tested before the agent's level is restored.
The chapter makes one claim of its own: an AI system that holds a delegated level on the blue side has to earn that level repeatedly, against cases whose answer someone other than the agent knows, and has to lose it quickly when it fails. The rest follows from that. The sections below say what to test, where the cases come from, how the stop path is exercised, how a provider's AI is tested through the contract, how a failure becomes a changed mandate and where testing in production must stop.
The temptation is to evaluate the model. The series has already argued that the usage mode and the delegation, not the model, set the risk (AI Red Team, Chapter 6) (90, Nordqvist, 2026), and a model benchmark says little about how a particular agent behaves with a particular mandate, connector set and case store. The unit of test is therefore the agent as deployed: model, prompts, tools, permissions, the data it reads, the policy decision point that bounds it and the people who review it. NIST's guide to security testing makes a related point about any testing programme: the purpose determines the technique, and testing carries its own risk of disruption, so it has to be planned (36, Scarfone et al., 2008).
Primary topics explored
- What to test in a defender's AI
- Seeded incidents from emulation
- Testing the stop path
- Testing a provider's AI through the contract
- Findings against agents and mandates
- Limits of testing in production
About AI Purple
AI Purple is Book 9 of The Transformed Architecture. It takes the loop between the red team and the blue team as its subject and asks what happens when AI sits on both sides of the table: the five stages of the loop (target, emulate, observe, improve, verify), rules of engagement and safe emulation, continuous emulation, the independence principle, coverage as a claim that needs evidence, the loop's own AI and how to defend it, operating models, decision rights, regulation, and the people involved, for European enterprises and public authorities.
It builds on the conclusions of Architecture as Code, AI Governance, AI Platform and Security, AI Red Team, AI Threat, The Transformed Security Architect, AI Boss and AI Blue.
Chief information security officers, heads of security operations, red-team leads, penetration testers, SOC managers, detection engineers, purple-team coordinators and incident commanders, security and AI architects, boards and management bodies, procurement leads, works councils and unions, data protection and audit functions, and public-sector leaders.
Gunnar Nordqvist
Conceived, directed and published the books of The Transformed Architecture.
Continue with the full chapter
Chapter 13: Testing the Defenders' AI: SOC Agents, Playbooks and Providers' AI under Emulation in AI Purple gives the detailed explanations, practical examples and implementation patterns behind this summary.
This whitepaper belongs to “The Loop Meets AI Systems” and is one of 33 drawn from the manuscript.