Every chapter of this book has ended its recommendations with a person: the unit manager who approves the roster, the recruiter who decides on the applicant, the team leader who can reassign the claim. This chapter asks when that person is a decider and when merely a signature on a system's output. It separates oversight of a system, which can be sampled, from the decision in an individual case, which cannot, and defines the oversight record and the oversight-quality metric. It concerns M3 Consequence, where every decision about a person has a named human decider, and M2 Evaluation, because evaluations feed consequences and measuring overseers is itself an evaluation; it touches M1 Direction, where rosters and allocations are approved at volume. It answers open question Q9: whether oversight can be real at hundreds of decisions an hour.
The chapter builds on conclusions of the earlier volumes: oversight is designed, not declared, approvals given in seconds signal rubber-stamping (F21) and stop rights are named in advance (F19) (AI Governance, Chapters 10, 13 and 21) (31, Nordqvist, 2026); approval quality is measured by time, rejections and reversals and reported in aggregate (F52; AI Red Team, Chapters 6 and 15) (33, Nordqvist, 2026); reliance on AI erodes the skills needed to catch its errors, answered by seeded-error drills and a team-level metric (F66; AI Threat, Chapters 6 and 14) (34, Nordqvist, 2026); and review must not start from the AI's answer (F76; The Transformed Security Architect, Chapter 14) (36, Nordqvist, 2026).
A requirement that a human be involved is the most common safeguard in algorithmic management, and the least tested. Ben Green surveyed 41 policies prescribing human oversight of government algorithms and found that people cannot perform the oversight the policies assume, so the policies legitimise faulty algorithms, provide a false sense of security and let vendors and agencies shirk accountability. He proposed institutional oversight instead, in which the organisation justifies both the algorithm and the form of oversight with evidence before adoption (158, Green, 2022).
Primary topics explored
- The human in the loop, examined
- Automation bias and selective adherence
- What makes oversight real
- Oversight of the system and the decision in the case
- The oversight record
- Measuring oversight quality
About AI Boss
AI Boss is Book 7 of The Transformed Architecture. It examines what happens when artificial intelligence takes the manager's chair: algorithmic management of direction, evaluation, consequence, development and care, the effects on people, the management of AI agents, Europe's legal answer, and how to design an accountable AI boss, for European enterprises and public authorities.
It builds on the conclusions of Architecture as Code, AI Governance, AI Platform and Security, AI Red Team, AI Threat and The Transformed Security Architect.
Executives, line managers, HR and people leaders, works councils and trade unions, public employers, architects and AI governance leads, data protection and legal officers, procurement and programme leaders, and risk, compliance and audit.
Gunnar Nordqvist
Conceived, directed and published the books of The Transformed Architecture.
Continue with the full chapter
Chapter 26: The Human in the Chair: Oversight That Is Real, Not Rubber-Stamped in AI Boss gives the detailed explanations, practical examples and implementation patterns behind this summary.
This whitepaper belongs to “Designing the Accountable AI Boss” and is one of 37 drawn from the manuscript.