The Limits of Data-Driven Decision Making

Data can discipline decisions. It can reveal a pattern that a confident anecdote misses, show whether a change worked for different groups, and make assumptions open to challenge.

A proxy is an observable measure used in place of a harder-to-measure goal. Wait time may approximate service quality, but it does not show whether a person received the right help. Proxies are useful when their limits stay in view.

The limitation is that a decision requires a question, and measurements inherit that framing. If a school tracks only test scores, it may overlook curiosity, student safety, or whether the measured skill transfers beyond the exam.

A metric is usually a proxy for something more valuable. Once people are rewarded for improving the proxy, they may change the number without improving the underlying outcome, especially when the measure is incomplete or easy to game.

Yet refusing measurement does not restore neutrality. It can leave decisions to habit, status, or intuition that is difficult to inspect. The answer is not fewer facts by default but a clearer account of what evidence can and cannot tell us.

Data can also average away local differences. A policy that helps most participants may make a service inaccessible to a smaller group, and an overall improvement can conceal who paid for it.

Averages describe a group, not every member of it. If a service improves overall but becomes much harder for people with limited internet access, the average can conceal a serious failure. Disaggregating results helps reveal distribution, though privacy and small samples need care.

Good decision processes pair measurement with context. They use multiple indicators, inspect outliers and unintended effects, hear from affected people, and revisit a choice when its assumptions no longer hold.

Imagine a help desk measured by how quickly each ticket closes. Staff may solve simple cases rapidly while difficult customers are passed between teams. Tracking resolution quality and repeat contact can reveal more of the experience, though those measures also need interpretation.

Causal claims need particular care. A change followed by improvement may have helped, but other events could explain the difference. Comparison groups, pre-defined outcomes, and transparent limits can make evidence more informative without making it infallible.

A usable inquiry lens asks what value the metric stands for, whose experience it misses, how incentives may reshape behavior, and what evidence would count against the preferred conclusion.

How can you make a decision accountable to evidence without letting the easiest number define what matters?