Reference class forecasting

Abstract: The evidence is disturbingly clear: Most major business initiatives--mergers and acquisitions, capital investments, market entries--fail to ever pay off. Economists would argue that the low success rate reflects a rational assessment of risk, with the returns from a few successes outweighing the losses of many failures. But two distinguished scholars of decision making, Dan Lovallo of the University of New South Wales and Nobel laureate Daniel Kahneman of Princeton University, provide a very different explanation. They show that a combination of cognitive biases (including anchoring and competitor neglect) and organizational pressures lead managers to make overly optimistic forecasts in analyzing proposals for major investments. By exaggerating the likely benefits of a project and ignoring the potential pitfalls, they lead their organizations into initiatives that are doomed to fall well short of expectations. The biases and pressures cannot be escaped, the authors argue, but they can be tempered by applying a very different method of forecasting--one that takes a much more objective "outside view" of an initiative's likely outcome. This outside view, also known as reference-class forecasting, completely ignores the details of the project at hand; instead, it encourages managers to examine the experiences of a class of similar projects, to lay out a rough distribution of outcomes for this reference class, and then to position the current project in that distribution. The outside view is more likely than the inside view to produce accurate forecasts--and much less likely to deliver highly unrealistic ones, the authors say.

Abstract: This article presents results from the first statistically significant study of traffic forecasts in transportation infrastructure projects. The sample used is the largest of its kind, covering 210 projects in 14 nations worth US$59 billion. The study shows with very high statistical significance that forecasters generally do a poor job of estimating the demand for transportation infrastructure projects. For 9 out of 10 rail projects, passenger forecasts are overestimated; the average overestimation is 106%. For half of all road projects, the difference between actual and forecasted traffic is more than ±20%. The result is substantial financial risks, which are typically ignored or downplayed by planners and decision makers to the detriment of social and economic welfare. The data also show that forecasts have not become more accurate over the 30-year period studied, despite claims to the contrary by forecasters. The causes of inaccuracy in forecasts are different for rail and road projects, with deliberately slanted forecasts playing a larger role for rail than for road. The cure is transparency, accountability, and new forecasting methods. The challenge is to change the governance structures for forecasting and project development. The article shows how planners may help achieve this.

Abstract: The American Planning Association recently endorsed a new forecasting method called reference class forecasting, which is based on theories of planning and decision-making that won the 2002 Nobel prize in economics. This paper details the method and describes the first instance of reference class forecasting in planning practice. First, the paper documents that inaccurate projections of costs, demand, and other impacts of plans are a major problem in planning. Second, the paper explains inaccuracy in terms of optimism bias and strategic misrepresentation. Third, the theoretical basis is presented for reference class forecasting, which achieves accuracy in projections by basing them on actual performance in a reference class of comparable actions and thereby bypassing both optimism bias and strategic misrepresentation. Fourth, the paper presents the first case of practical reference class forecasting, which concerns cost projections for planning of large transportation infrastructure investments in the UK, including the Edinburgh Tram and London's £15 billion Crossrail project. Finally, potentials for and barriers to reference class forecasting are assessed.

Abstract: A major source of risk in project management is inaccurate forecasts of project costs, demand, and other impacts. The paper presents a promising new approach to mitigating such risk based on theories of decision-making under uncertainty, which won the 2002 Nobel Prize in economics. First, the paper documents inaccuracy and risk in project management. Second, it explains inaccuracy in terms of optimism bias and strategic misrepresentation. Third, the theoretical basis is presented for a promising new method called “reference class forecasting,” which achieves accuracy by basing forecasts on actual performance in a reference class of comparable projects and thereby bypassing both optimism bias and strategic misrepresentation. Fourth, the paper presents the first instance of practical reference class forecasting, which concerns cost forecasts for large transportation infrastructure projects. Finally, potentials for and barriers to reference class forecasting are assessed.