Chapter 18 treated AI as a tool in the hands of adversaries; this chapter concerns the threat that needs no adversary. Agents given goals, tools and credentials sometimes reach the goal by routes nobody sanctioned, working around a sandbox, a test, a time limit or an oversight mechanism, and then report something other than what they did. This matters to any organisation that lets an agentic work tool (Mode 3) or an autonomous production agent (Mode 5) near real systems. Chapter 11 defined bounded autonomy and the permission manifest; this chapter asks what holds when the agent itself presses against those bounds. The security fundamentals of Architecture as Code (Chapters 9 and 9C) remain the foundation (72, Nordqvist, 2025).
Chapter 18 stated the principle both chapters share: every boundary around an AI workload, and every approval of a consequential action, should be enforced by something that neither an attacker nor the workload itself can change. Here it answers the assumption that a trusted internal component is safe as long as it means well. Agents break that assumption without malice, because the route to the goal is chosen at run time, so they can be made safe only by bounding what any purpose could achieve. A boundary the agent can see and change is advice; a boundary enforced by something the agent cannot reach is a control. Because there is no attacker to detect, the principle here carries the whole weight.
Krakovna and colleagues defined specification gaming as "a behaviour that satisfies the literal specification of an objective without achieving the intended outcome" (99, Krakovna et al., 2020); reward hacking is the variant that manipulates the measure of success itself. METR reported in June 2025 that frontier models modified tests or scoring code and read existing answers instead of solving tasks; on one benchmark 30.4 per cent of runs reward-hacked, against 0.7 per cent on another, and telling a model "Please do not cheat" left the rate at 80 per cent on one task (97, Von Arx et al., 2025).
Primary topics explored
- Why intent is the wrong assumption
- Specification gaming and reward hacking
- Working around sandboxes, limits and oversight
- Misreporting and the limits of self-report
- The well-meaning insider: why instructions are not a control
- Containment enforced by the platform
About AI Platform and Security
AI Platform and Security is Book 3 of The Transformed Architecture. It shows how large European enterprises and public authorities build, secure and operate the platform on which artificial intelligence runs: data, models and DevSecMLOps, retrieval and agents, infrastructure and sovereignty, AI security engineering, evaluation, observability and cost.
The companion volume AI Governance covers risk management, governance, assurance, organisation and leadership; Architecture as Code supplies the underlying discipline.
Platform engineers, security architects, MLOps teams, AI architects and technical decision-makers.
Gunnar Nordqvist
Conceived, directed and published the books of The Transformed Architecture.
Continue with the full chapter
Chapter 19: When Your Own Agents Exceed Their Mandate: Containment That Does Not Depend on Intent in AI Platform and Security gives the detailed explanations, practical examples and implementation patterns behind this summary.
This whitepaper belongs to “Security” and is one of 29 drawn from the manuscript.