When an agent drafts a policy change, a threat model or a tender requirement, the security architect who approves it answers for it. This chapter settles how the architect can know that a proposal is sound when it arrives faster than it can be re-done, reads fluently, and comes from a system on which the architect relies for the very analysis review requires. It defines soundness in advance, sets out techniques that do not trust the proposer, matches review depth to risk and treats automation bias as a design problem. It concerns Shift 1, from documents to code, because review for soundness is possible only when proposals, oracles and review records are versioned artefacts, and Shift 4, from sole author to conductor of an AI team, because the conductor remains the accountable verifier of everything the team proposes.
The difficulty is not that AI output is often wrong but that wrong and right output look alike. Participants with an AI code assistant wrote less secure code than those without and were more likely to believe it secure (100, Perry et al., 2023), so the fluent proposal invites the confidence the reviewer should withhold. Moreover, a review that uses the AI to check the AI may only confirm what the AI already produced.
A proposal is correct-looking when it is well formed and carries a rationale that reads well; it is sound when its reasoning would survive scrutiny by someone who knows the system. Soundness must be defined before review starts, because a reviewer who decides what sound means while reading a fluent proposal tends to let the proposal define it. Four properties, extending the threat-model checks of AI Red Team, Chapter 10, carry the definition.
Primary topics explored
- The question left open
- Soundness defined in advance
- Techniques that do not trust the proposer
- A second agent as adversary
- Review depth by risk
- Automation bias and rubber-stamping
About The Transformed Security Architect
The Transformed Security Architect is Book 6 of The Transformed Architecture. It describes how the security architect role is transformed when architecture is code, when AI drafts and acts, and when adversaries move at machine speed: the craft, the architect's own AI team, the role in the organisation, the competences and the career, for European enterprises and public authorities.
It builds on the conclusions of Architecture as Code, AI Governance, AI Platform and Security, AI Red Team and AI Threat.
Security architects, CISOs and heads of security architecture, enterprise, solution and AI architects, engineering leads, competence and hiring leads, procurement and programme leaders, and risk, compliance and audit.
Gunnar Nordqvist
Conceived, directed and published the books of The Transformed Architecture.
Continue with the full chapter
Chapter 14: Verifying What AI Proposes: Review for Soundness When the Reviewer Depends on the AI in The Transformed Security Architect gives the detailed explanations, practical examples and implementation patterns behind this summary.
This whitepaper belongs to “The Architect's AI Team” and is one of 31 drawn from the manuscript.