Chapter 12 turned the red team on the organisation's estate. This chapter turns it on the organisation's own AI: assistants, the retrieval systems behind them and agents that act with identities of their own. These systems fail in ways conventional penetration testing does not reach: an assistant can be steered by a document it was asked to summarise, an agent can use a legitimate tool for an illegitimate purpose, and a model upgrade can quietly undo a guardrail. The chapter therefore shows how to scope the compound system, use taxonomies as test plans, turn campaigns into gating suites, test agent containment, and judge how far testing can be trusted.
In CSF terms the chapter is responsible for two Categories of the Identify Function (19, NIST, 2024). ID.RA Risk Assessment asks that vulnerabilities are identified, validated and recorded (ID.RA-01), threats identified (ID.RA-03), likelihoods and impacts understood (ID.RA-04, ID.RA-05) and risk responses chosen and tracked (ID.RA-06). ID.IM Improvement asks that improvements are identified from evaluations (ID.IM-01) and from security tests and exercises, including those run with suppliers (ID.IM-02). The chapter treats both Categories through the three focus areas of the NIST Cyber AI Profile (20, NIST, 2025), with securing AI systems as its core.
A deployed AI application is a compound system of users and content sources, a gateway, prompts and the model, a retrieval store, an agent with a permission manifest, and tools, as Figure 13.1 shows. Most failures that matter arise where two parts meet: a model that refuses to disclose personal data when asked may still reproduce it when retrieval places a case document in its context.
Primary topics explored
- Scoping the compound system
- Taxonomies as test plans
- Suites, harnesses and gates
- Red teaming agents
- The limits of testing
- Red teaming AI across modes, levels and focus areas
About AI Red Team
AI Red Team is Book 4 of The Transformed Architecture. It shows how large European enterprises and public authorities put artificial intelligence to work as an active part of cyber defence: AI-assisted red teaming and adversary emulation, red teaming of AI systems themselves, the AI-assisted security operations centre, detection engineering, threat intelligence, incident response and purple teaming, all under a lawful mandate and with evidence a supervisor can inspect.
It builds on the conclusions of Architecture as Code, AI Governance and AI Platform and Security.
CISOs, SOC leads, red and blue team practitioners, security architects, AI architects and technical decision-makers.
Gunnar Nordqvist
Conceived, directed and published the books of The Transformed Architecture.
Continue with the full chapter
Chapter 13: Red Teaming AI Systems and Agents in AI Red Team gives the detailed explanations, practical examples and implementation patterns behind this summary.
This whitepaper belongs to “Identify” and is one of 32 drawn from the manuscript.