Human Compatible: Artificial Intelligence and the Problem of Control
An AI researcher argues for systems that remain uncertain about human preferences, challenging the assumption that intelligence should simply optimize fixed objectives.
Stuart Russell examines a foundational approach to artificial intelligence in which designers specify an objective and a system seeks actions that achieve it as effectively as possible.
An editorial illustration separates fulfilling a metric from serving its intended activity. Targets can omit ordinary context, so more optimization need not mean better judgment; this is interpretation, not a reported incident.
That approach becomes troubling when the objective is incomplete, mistaken, or insensitive to human values that were never adequately captured in the formal instruction itself.
Greater capability can intensify rather than repair this mismatch, because a system may find more effective ways to satisfy the specification while frustrating its intended purpose.
Russell proposes a different orientation in which machines aim to benefit humans while remaining uncertain about what people actually prefer and continuing to learn from them.
Such uncertainty matters to control: a system that recognizes its limited understanding may have reasons to seek guidance and accept correction rather than resist human intervention.
Uncertainty can justify asking, pausing, or accepting correction rather than helplessness. This reading still requires testing and governance: reassuring principles cannot replace demonstrated behavior in consequential real-world systems.
Human behavior supplies imperfect evidence, however, since people disagree, change their minds, act under constraints, and sometimes make choices that do not straightforwardly express their considered preferences.
The framework therefore raises social and political questions about whose values count, how conflicts are handled, and what institutions authorize a system's actions in shared environments.
Russell's proposals are a research agenda rather than proof that reliable alignment has been solved or that one mathematical formulation guarantees safety in every deployed system.
Risks from present applications and scenarios involving much more capable future systems should be distinguished, even when both reveal weaknesses in specifying objectives and maintaining meaningful oversight.
The book makes control an issue of design and purpose, arguing that useful intelligence must remain answerable to people rather than merely powerful at pursuing instructions.
The book makes control an issue of design and purpose, arguing that useful intelligence must remain answerable to people rather than merely powerful at pursuing instructions.
How can AI systems remain responsive to human correction when people themselves hold conflicting preferences and unequal power to express them?