Cognitive Reflection Test

Abstract: This paper introduces a three-item "Cognitive Reflection Test" (CRT) as a simple measure of one type of cognitive ability—the ability or disposition to reflect on a question and resist reporting the first response that comes to mind. The author will show that CRT scores are predictive of the types of choices that feature prominently in tests of decision-making theories, like expected utility theory and prospect theory. Indeed, the relation is sometimes so strong that the preferences themselves effectively function as expressions of cognitive ability—an empirical fact begging for a theoretical explanation. The author examines the relation between CRT scores and two important decision-making characteristics: time preference and risk preference. The CRT scores are then compared with other measures of cognitive ability or cognitive "style." The CRT scores exhibit considerable difference between men and women and the article explores how this relates to sex differences in time and risk preferences. The final section addresses the interpretation of correlations between cognitive abilities and decision-making characteristics.

Abstract: Much research in the last two decades has demon- strated that human responses deviate from the performance deemed normative according to various models of decision mak- ing and rational judgment (e.g., the basic axioms of utility theory). This gap between the normative and the descriptive can be inter- preted as indicating systematic irrationalities in human cognition. However, four alternative interpretations preserve the assumption that human behavior and cognition is largely rational. These posit that the gap is due to (1) performance errors, (2) computational limitations, (3) the wrong norm being applied by the experi- menter, and (4) a different construal of the task by the subject. In the debates about the viability of these alternative explanations, attention has been focused too narrowly on the modal response. In a series of experiments involving most of the classic tasks in the heuristics and biases literature, we have examined the implica- tions of individual differences in performance for each of the four explanations of the normative/descriptive gap. Performance er- rors are a minor factor in the gap; computational limitations un- derlie non-normative responding on several tasks, particularly those that involve some type of cognitive decontextualization. Un- expected patterns of covariance can suggest when the wrong norm is being applied to a task or when an alternative construal of the task should be considered appropriate.

Abstract: The program of research now known as the heuristics and biases approach began with a survey of 84 participants at the 1969 meetings of the Mathematical Psychology Society and the American Psychological Association (Tversky & Kahneman, 1971). The respondents, including several authors of statistics texts, were asked realistic questions about the robustness of statistical estimates and the replicability of research results. The article commented tongue-in-heek on the prevalence of a belief that the law of large numbers applies to small numbers as well: Respondents placed too much confidence in the results of small samples, and their statistical judgments showed little sensitivity to sample size. The mathematical psychologists who participated in the survey not only should have known better – they did know better. Although their intuitive guesses were off the mark, most of them could have computed the correct answers on the back of an envelope. These sophisticated individuals apparently had access to two distinct approaches for answering statistical questions: one that is spontaneous, intuitive, effortless, and fast; and another that is deliberate, rule-governed, effortful, and slow. The persistence of large biases in the guesses of experts raised doubts about the educability of statistical intuitions. Moreover, it was known that the same biases affect choices in the real world, where researchers commonly select sample sizes that are too small to provide a fair test of their hypotheses (Cohen, 1969, 1992).

Abstract: The Cognitive Reflection Test (CRT; Frederick, 2005) is designed to measure the tendency to override a prepotent response alternative that is incorrect and to engage in further reflection that leads to the correct response. In this study, we showed that the CRT is a more potent predictor of performance on a wide sample of tasks from the heuristics-and-biases literature than measures of cognitive ability, thinking dispositions, and executive functioning. Although the CRT has a substantial correlation with cognitive ability, a series of regression analyses indicated that the CRT was a unique predictor of performance on heuristics-and-biases tasks. It accounted for substantial additional variance after the other measures of individual differences had been statistically controlled. We conjecture that this is because neither intelligence tests nor measures of executive functioning assess the tendency toward miserly processing in the way that the CRT does. We argue that the CRT is a particularly potent measure of the tendency toward miserly processing because it is a performance measure rather than a self-report measure.