In today’s data-driven world, organizations increasingly rely on data to inform their decisions. However, as the Harvard Business Review article “Where Data-Driven Decision-Making Can Go Wrong” emphasizes, this reliance can lead to significant pitfalls. By acknowledging these challenges and connecting them to core themes I’ve introduced in previous posts, like “impact paths,” “decision black holes,” and “data shoehorning,” we can transform how data informs decisions and avoid common traps.

Impact Paths: Defining Clear Outcomes to Avoid Data Pitfalls

I describe impact paths as the clear and intentional pathways that connect data insights to actionable decisions and measurable outcomes. When impact paths are well-defined, data flows seamlessly from analysis to action, ensuring that insights drive meaningful change. However, when these pathways are poorly defined—or absent altogether—insights risk falling into decision black holes, where they fail to inform decisions or achieve organizational relevance.

The Importance of Defining Impact Paths

The HBR article emphasizes the need for rigorous evaluation of evidence to ensure it applies to the decision context. As the authors state:

“Taking the time to discuss the nuances of analyses—including sample size, composition, outcomes being measured, and the approach to separating causation from correlation—is vital.”

For impact paths to lead to actionable insights, leaders must ensure that analyses are both internally valid (answering the right question) and externally valid (generalizing appropriately).¹

• Internal validity ensures that your analysis accurately answers the specific question it was designed to address. For instance, when evaluating the impact of a marketing campaign, internal validity would involve isolating the effect of the campaign itself from other variables, like seasonal trends or concurrent promotions.
• External validity asks whether the findings apply beyond the specific context of the analysis. For example, a tactic that succeeds in a small pilot region might fail to scale due to differences in demographics or market conditions.

By rigorously evaluating both dimensions, leaders create a foundation for actionable decisions that are both precise and broadly applicable. Without this rigor, even seemingly strong data can fail to deliver meaningful impact.

The Risk of Decision Black Holes and the Importance of Impact Paths

Without clear impact paths, data often falls into decision black holes—where valuable insights are collected but fail to inform decisions or properly align with broader organizational priorities. The HBR article describes this as a “disconnect between what is measured and what matters.” As the authors note:

“You must ensure that you’re measuring an outcome that really matters instead of one that is simply easy to measure. And you need to look for—or undertake—other research that might confirm or contradict the evidence.”

Take customer journey analysis as an example. If a marketing team focuses solely on easily measurable metrics like impressions or click-through rates, they risk missing critical insights into long-term retention or progression through touchpoints. These broader insights, while harder to measure, are often more aligned with strategic goals.

The connection between impact paths and decision black holes is clear: poorly defined or absent impact paths are the primary reason valuable insights fail to influence decisions. Leaders can avoid these pitfalls by ensuring all metrics and insights directly support organizational objectives. Key questions to guide this process include:

• “Do these metrics reflect broader organizational goals?”
• “Are we capturing both intended and unintended consequences of our actions?”
• “Are we looking at outcomes over a sufficient time frame to understand long-term impacts?”

By rigorously defining impact paths and addressing these questions, organizations can keep data connected to actionable decisions and avoid the irrelevance of decision black holes. When leaders prioritize nuanced discussions of causation, validity, and relevance—like those emphasized in the HBR article—they ensure data remains actionable and directly tied to the organization’s strategic vision.

Data Shoehorning: Forcing Data to Fit the Narrative

Data shoehorning refers to a type of narrative engineering that involves the deliberate construction of a story using data, even when the supposed connections are tenuous or misleading. This approach often stems from cognitive biases like confirmation bias—seeking out evidence that supports a desired conclusion while ignoring data that might contradict it.and undermines the integrity of decision-making.

The HBR article highlights the dangers of this practice:

“Too often predetermined beliefs, problematic comparisons, and groupthink dominate discussions. Research from psychology and economics suggests that biases—such as base rate neglect, the tendency to overlook general statistical information in favor of specific case information or anecdotes, and confirmation bias, the propensity to seek out and overweight results that support your existing beliefs—also hinder the systematic weighing of evidence.”

The HBR article offers a striking example from eBay. A consulting report concluded that search engine advertising increased sales, based on higher purchase values in markets with more ad spend. However, a deeper experiment revealed that the correlation was spurious: the ads were simply targeting customers already likely to shop on eBay, or coincided with natural spikes in demand.

This example highlights the risk of confusing correlation with causation—a hallmark of data shoehorning. To guard against this, organizations should rigorously evaluate evidence by:

• Asking whether causation has been clearly established.
• Investigating confounding variables that might explain observed patterns.
• Prioritizing experiments or other studies to uncover causal relationships.

By letting the data guide the story—not the other way around—leaders can ensure their decisions are grounded in reality rather than bias.

Key Takeaways for Decision Scientists

The HBR article offers valuable lessons for decision scientists that reinforce the importance of these concepts:

1. Define and Strengthen Impact Paths:
   • Link insights to decisions by systematically assessing their validity and relevance.
   • Clear impact paths ensure that data leads to decisions by bridging the gap between analysis and action.
2. Avoid Decision Black Holes:
   • Ensure insights are tied to meaningful outcomes and organizational goals.
   • Avoiding decision black holes means focusing on the “so what” of metrics—ensuring they capture what matters most to the organization rather than what’s easiest to measure.
3. Guard Against Data Shoehorning:
   • Let the data guide the story—not the other way around.
   • Preventing shoehorning requires asking tough questions about methodology, confounding factors, and alternative explanations to ensure the integrity of the insights.

Conclusion

Decision-making is inherently complex, and no dataset is a panacea. However, by defining impact paths, avoiding decision black holes, and resisting the temptation to shoehorn data, organizations can make better decisions, period. As the HBR article notes, “Employing a systematic approach to the collection, analysis, and interpretation [of data]” is critical to reaping its full benefits.

By following these principles, decision scientists can ensure that data doesn’t just inform decisions—it transforms them into actions that drive meaningful, measurable outcomes.

1. Where Data-Driven Decision-Making Can Go Wrong ↩︎