Security awareness has long meant an annual e-learning module, a quarterly simulated phishing email and a click-rate chart, and AI has broken the assumptions behind that routine. A lure no longer betrays itself through clumsy grammar, a voice on the telephone can belong to nobody, and a video call can show a colleague who is not there. Meanwhile, the people inside the security function have a new task that no awareness programme prepared them for: supervising AI that triages alerts, drafts detections and proposes containment, which they must learn to check, override and, where it acts on its own, stop.
This chapter therefore treats two disciplines that AI has drawn together: resilience to AI-generated deception, built into processes and tested on processes rather than individuals, and AI literacy for security staff, so that the people who work with security AI keep the judgement it is supposed to support. In CSF terms the chapter is responsible for PR.AT Awareness and Training, whose outcome is that personnel receive awareness and training to perform their cybersecurity-related tasks. PR.AT-01 covers general personnel and PR.AT-02 individuals in specialised roles (19, NIST, 2024). With security AI in the estate, both audiences change: general staff face AI-generated deception, and specialised roles now include those who supervise AI.
The chapter builds on two earlier conclusions. AI Governance, Chapters 10 and 21, concluded that human oversight must be designed, not declared, and that approvals given in seconds signal rubber-stamping (72, Nordqvist, 2026). AI Platform and Security, Chapter 18, committed to identity and approvals that resist synthetic media and to AI for defence under oversight, where human verification is the bottleneck (73, Nordqvist, 2026). Training is how the organisation makes both commitments real.
Primary topics explored
- Deception at scale
- Test the process, not the person
- Measuring resilience
- Training analysts for AI-assisted work
- Summary
- Sources
About AI Red Team
AI Red Team is Book 4 of The Transformed Architecture. It shows how large European enterprises and public authorities put artificial intelligence to work as an active part of cyber defence: AI-assisted red teaming and adversary emulation, red teaming of AI systems themselves, the AI-assisted security operations centre, detection engineering, threat intelligence, incident response and purple teaming, all under a lawful mandate and with evidence a supervisor can inspect.
It builds on the conclusions of Architecture as Code, AI Governance and AI Platform and Security.
CISOs, SOC leads, red and blue team practitioners, security architects, AI architects and technical decision-makers.
Gunnar Nordqvist
Conceived, directed and published the books of The Transformed Architecture.
Continue with the full chapter
Chapter 15: Awareness and Training: Resilience to AI-Generated Deception in AI Red Team gives the detailed explanations, practical examples and implementation patterns behind this summary.
This whitepaper belongs to “Protect” and is one of 32 drawn from the manuscript.