A worker who believes that a score, a route or a pay input about her is wrong needs three things: to learn what the system did and who decided, to reach a person who can change the outcome, and to see the error corrected in her own case and in the system that produced it. This chapter designs those three things into the AI boss. It concerns M2 Evaluation, where scores and ratings are contested, and M3 Consequence, where the pay, promotion, discipline and access to work that follow from them are contested; it also reaches M1 Direction and M5 Care where a contested instruction, such as a route with no time between two visits, turns out to be a risk to health. It introduces the contestation record (CNT-*), the explanation a person receives and the appeal path with its time limits. Its argument is that an appeal reaching a person able to change the outcome is the condition on which every degree of delegation above Level 1 Assisted depends.
Contestability has three justifications. The first is procedural justice. Lee's experiment found algorithmic decisions on tasks requiring human skills perceived as less fair and less trustworthy than the same decisions attributed to a manager (146, Lee, 2018), and Binns and colleagues found that people apply the same dimensions of justice to algorithmic decisions as to human ones, with no single style of explanation best across scenarios (145, Binns et al., 2018). Part of what people judge is whether a person will answer.
The second is accountability. Bovens describes it as a relationship in which an actor must explain and justify its conduct to a forum that can question it and pass judgement, with consequences for the actor (107, Bovens, 2007). A contest is that relationship at the scale of one decision, and a system cannot take the actor's place, which is why the rule that every M3 Consequence decision has a named human decider is also the precondition of contest.
Primary topics explored
- Why contestability
- What a worker can learn
- Explanations people can use
- The appeal path
- The contestation record
- Correcting the decision and the system
About AI Boss
AI Boss is Book 7 of The Transformed Architecture. It examines what happens when artificial intelligence takes the manager's chair: algorithmic management of direction, evaluation, consequence, development and care, the effects on people, the management of AI agents, Europe's legal answer, and how to design an accountable AI boss, for European enterprises and public authorities.
It builds on the conclusions of Architecture as Code, AI Governance, AI Platform and Security, AI Red Team, AI Threat and The Transformed Security Architect.
Executives, line managers, HR and people leaders, works councils and trade unions, public employers, architects and AI governance leads, data protection and legal officers, procurement and programme leaders, and risk, compliance and audit.
Gunnar Nordqvist
Conceived, directed and published the books of The Transformed Architecture.
Continue with the full chapter
Chapter 27: Contestability by Design: Explanation, Appeal and Correction in AI Boss gives the detailed explanations, practical examples and implementation patterns behind this summary.
This whitepaper belongs to “Designing the Accountable AI Boss” and is one of 37 drawn from the manuscript.