The watching state is the oldest political fear about machines. In Nineteen Eighty-Four the telescreen both broadcasts and observes, and nobody knows when they are being watched (37, Orwell, 1949). Orwell's Party still needed human informers and clerks to turn observation into judgement, and that labour limited surveillance for most of the twentieth century. AI removes much of that cost, because a camera network, a mailbox or a payment ledger can now be read, linked and scored by software almost for free.
This chapter tests T7 The watching state, which Chapter 2 widened beyond the state following Shoshana Zuboff's argument that the defining surveillance of this century is commercial (170, Zuboff, 2019). Few European organisations set out to build a watching machine, but many build pieces of one: an inventory that keeps everything, a security platform that reads every message, an analytics feature that links data collected for one purpose to decisions taken for another.
Table 13.1: Uses along the scale.
Primary topics explored
- Surveillance capitalism and the state
- Facial recognition and scraping
- Censorship built into models
- Article 5 prohibitions
- The employee and the SOC
- The debate
About AI Threat
AI Threat is Book 5 of The Transformed Architecture. It investigates whether the dystopias of science fiction, with AI as the threat, are coming true. It covers every scale of the threat, from the human mind and society to misuse, machines beyond control and the end of the world, stages every debate constructively, and turns the verdict into what individuals, European enterprises and public authorities can do.
It builds on the conclusions of Architecture as Code, AI Governance, AI Platform and Security and AI Red Team.
Boards, chief risk officers, CISOs, AI governance leads, architects and public-sector leaders.
Gunnar Nordqvist
Conceived, directed and published the books of The Transformed Architecture.
Continue with the full chapter
Chapter 13: Surveillance, Profiling and the Erosion of Privacy in AI Threat gives the detailed explanations, practical examples and implementation patterns behind this summary.
This whitepaper belongs to “Society and Truth” and is one of 39 drawn from the manuscript.