Picture the evaluation meeting for a rostering system at Nordvik Region, the fictional regional public authority that runs through this book as an illustration. The tender is for software that will draft four-week rosters for hospital wards and visit routes for home care, and the questionnaire sent to bidders contains a line that many procurements contain: "Is the system automated?" The bidder has ticked yes. The union representative at the table reads the tick and hears that a machine will decide who works when. The head of home care hears that unit managers will be spared a weekend of spreadsheets. The data protection officer reaches for Article 22 of the GDPR, which concerns decisions based solely on automated processing (41, European Union, 2016).
All three readings are reasonable, and all three miss, because the question cannot be answered with a tick. The system that was procured, drawn in this book as the unit NR-SW, drafts a roster that a unit manager must approve before anyone sees it. It flags a breach of the rest rules and leaves the manager to resolve it. And it can offer an open shift to staff who have opted in; whether it may do so without asking anyone first is what the Region and the unions have to settle in writing. Three actions, three different answers to what the machine does on its own.
"Is it automated?" is the most natural question in the world, and the landscape does not divide that way. In my reading, and the chapter says below how far the data go, much AI at work drafts something for a person who then decides; some acts inside written limits; a little acts without anyone approving each step; and one system can do all three on the same afternoon. This chapter is about the instinct that hides this, the evidence, and the two views of the notation that make it visible.
Primary topics explored
- The questions
- The instinct
- The evidence
- The picture corrected
- The rule of thumb
- What to do
About Landscape
Landscape is a companion to The Transformed Architecture. It examines what happens when artificial intelligence takes the manager's chair: algorithmic management of direction, evaluation, consequence, development and care, the effects on people, the management of AI agents, Europe's legal answer, and how to design an accountable AI boss, for European enterprises and public authorities.
It builds on the conclusions of Architecture as Code, AI Governance, AI Platform and Security, AI Red Team, AI Threat and The Transformed Security Architect.
Executives, line managers, HR and people leaders, works councils and trade unions, public employers, architects and AI governance leads, data protection and legal officers, procurement and programme leaders, and risk, compliance and audit.
Gunnar Nordqvist
Conceived, directed and published the books of The Transformed Architecture.
Continue with the full chapter
Chapter 6: The Binary Instinct: Neither Automated Nor Manual in Landscape gives the detailed explanations, practical examples and implementation patterns behind this summary.
This whitepaper belongs to “The Instincts” and is one of 26 drawn from the manuscript.