Decline effect

Abstract: Aspirited debate is occurring in the popular press on social science and the scientific method. Jonah Lehrer (2010b) published an article in the New Yorker whose primary topic was the "decline effect," where initial estimates of interventions' effectiveness weaken when replicated. Lehrer's examples came from pharmacology, medicine, psychology, zoology, and more. A debate was ignited, largely on the Internet (for example, "Neurological blog," "Respectful Insolence," "Science Based Medicine," "Psychology Today," "ABC News"). In the next issue of the New Yorker, Lehrer (2011) responded to letters and e-mails in another article as well as an article in Wired (Lehrer, 2010a). Issues raised by the debate deserve consideration by social work researchers. This editorial explains the decline effect and presents comments on replications, failure to submit, publication bias, and "a fundamental cognitive flaw" (Lehrer, 2010b). The term "decline effect" has not appeared in social work literature, nor have estimates of the number of replications, the extent of failing to report research, or evidence of journals' publication bias, with the exception of Dickersin (1997). I base my comments on almost 30 years in academic social work at four universities, extensive reading of our research literature, teaching research, reviewing manuscripts, and publishing. THE DECLINE EFFECT The "decline effect" is a phrase attributed to Rhine (1938) in research on extrasensory perception describing the situation in which a research participant guessed a hidden card at a rate beyond chance. Alter repeated testing, however, "the effect dramatically diminished" (Rhine, 1938). Lehrer (2010b) reported a similar pattern in research about antipsychotic medications. Early clinical trials documented the medications' positive impact on psychiatric symptoms, but recent research indicates that "the therapeutic power of the drugs appeared to be steadily waning" (Lehrer, 2010b, para. 2). Another researcher, Jonathon Schooler, attempted to replicate Rhine's finding of a decline effect by studying precognition. At first, Schooler found higher than expected precognition, "'but then, as we kept on running subjects, the effect size'--a standard statistical measure--'kept getting smaller and smaller'" (Lehrer, 2010b, p. 6-7). Lehrer (2010b) described similar results in research on language and memory, zoology, biology, and epidemiology. He described a study in which the researcher identified the 49 most frequently cited clinical research studies and reviewed the subsequent research literature to see if there might be a decline effect (Ioannidis, 2005).The sampled studies had to have 1,000 or more citations. In 45 of the sampled studies the intervention in question was found to be effective. (It is interesting to note that the four studies in the sample that showed no efficacy for the intervention being studied were contradicting earlier claims of efficacy). Thirty-four of the studies in the sample had been replicated: 21% were contradicted by subsequent research, and 21% had weaker effects. One should not conclude, Ioannidis (2005) pointed out, "that the original studies were totally wrong and the newer ones are correct simply because they are larger or better controlled" (p. 224). To what degree has the decline effect been observed in social work? A literature search reveals no reference to it as such. The prior question, however, and the one of central concern here is this: How frequently are replication studies done in social work? Without replications, decline effects cannot be observed. Replication studies seem to be fairly rare in social work; if they occur, it is likely to happen with more heavily funded and broadly implemented interventions. Failure to replicate intervention studies is costly for reasons far beyond documenting decline effects: Replication, widely acknowledged as the cornerstone of science, brings many benefits to the discipline that embraces it. …

Abstract: The Scientific Method and the Politics of TruthMedical Research for Hire: The Political Economy of Pharmaceutical Clinical Trials. Jill A. Fisher. New Brunswick, NJ: Rutgers University Press, 2009. Paperback ed. 257 pages. ISBN 9780813544106. $65.00, (PB) $25.95The Professional Guinea Pig: Big Pharma and the Risky World of Human Subjects. Roberto Abadie. Durham, NC: Duke University Press, 2010. Paperback ed. 184 pages. ISBN 9780822348238. $79.95, (PB) $22.95When Experiments Travel: Clinical Trials and the Global Search for Human Subjects. Adriana Petryna. Princeton, NJ: Princeton University Press, 2009. Paperback ed. 258 pages. ISBN 978069112657-9. $70.00, (PB) $27.95In late 2010, the issue of truth and falsehood in experiment-based scientific research-especially though not exclusively medical research-came to the forefront in two articles by David H. Freedman and Jonah Lehrer published in The Atlantic Monthly and The New Yorker, respectively.1 Freedman focused on the work of John P.A. Ioannidis, describing him as "one of the world's foremost experts on the credibility of medical research." Ioannidis, the author of articles with titles such as "Why Most Published Research Findings Are False" and "Contradicted and Initially Stronger Effects in Highly Cited Clinical Research," was finally receiving the mainstream attention that he deserved.2 As Freedman noted, Ioannidis's central contention that "as much as 90 percent of ... published medical information" is "misleading, exaggerated, and often flat-out wrong" has been "widely accepted by the medical community," yet it has had no real effect on the way that health researchers go about their business.In Ioannidis's opinion, the underlying problem is a simple, yet intractable one: "There is an intellectual conflict of interest that pressures researchers to find whatever it is that is most likely to get them funded" and published in the best professional journals.3 Funding agencies and journals almost always favor proposals and studies that give (or have the potential to give) positive results (i.e., statistically significant outcomes that validate the researcher's hypothesis). In turn, funding and publication in top journals are key factors in determining whether researchers receive tenure and/or progress in their careers. As a result, researchers "headed into their studies wanting certain results-and, lo and behold, they were getting them" based on a surprising number of errors that ranged "from what questions [they] posed, to how they set up the studies, to which patients they recruited for the studies, to which measurements they took, to how they analyzed the data, to how they presented their results, to how particular studies came to be published in medical journals."4 It is therefore not surprising that many "claimed research findings may often be simply accurate measures of [a] prevailing bias," where bias includes, but is not limited to, circumstances "where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more [research] teams are involved in a scientific field in chase of statistical significance."5The issue of "an intellectual conflict of interest" is not limited to medical science; it can be found in psychology, ecology, zoology, and evolutionary biology, not to mention anthropology and economics. As Lehrer documents, it often manifests itself as something called the "decline effect," whereby initially dramatic experimental results become markedly less dramatic and even nonexistent over time, most likely due to the phenomenon of regression to the mean. In other words, the initial positive results that, in a single experimental instance, confirmed a researcher's hypothesis cannot be replicated. The law of averages has taken over: another group of subjects (i.e., participants in the experiments) produced vastly different outcomes when asked to perform the very same experimental protocol as the initial group of subjects. …

Abstract: Empirical sciences in general and psychological science in particular are plagued by replicability problems and biased published effect sizes. Although dissemination bias-related phenomena such as publication bias, time-lag bias, or visibility bias are well-known and have been intensively studied, another variant of effect distorting mechanisms, so-called decline effects, have not. Conceptually, decline effects are rooted in low initial (exploratory) study power due to strategic researcher behavior and can be expected to yield overproportional effect declines. Although decline effects have been documented in individual meta-analytic investigations, systematic evidence for decline effects in the psychological literature remains to date unavailable. Therefore, we present in this meta-meta-analysis a systematic investigation of the decline effect in intelligence research. In all, data from 22 meta-analyses comprising 36 meta-analytical and 1,391 primary effect sizes (N = 697,000+) that have been published in the journal Intelligence were included in our analyses. Two different analytic approaches showed consistent evidence for a higher prevalence of cross-temporal effect declines compared to effect increases, yielding a ratio of about 2:1. Moreover, effect declines were considerably stronger when referenced to the initial primary study within a meta-analysis, yielding about twice the magnitude of effect increases. Effect misestimations were more substantial when initial studies had smaller sample sizes and reported larger effects, thus indicating suboptimal initial study power as the main driver of effect misestimations in initial studies. Post hoc study power comparisons of initial versus subsequent studies were consistent with this interpretation, showing substantially lower initial study power of declining, than of increasing effects. Our findings add another facet to the ever accumulating evidence about non-trivial effect misestimations in the scientific literature. We therefore stress the necessity for more rigorous protocols when it comes to designing and conducting primary research as well as reporting findings in exploratory and replication studies. Increasing transparency in scientific processes such as data sharing, (exploratory) study preregistration, but also self- (or independent) replication preceding the publication of exploratory findings may be suitable approaches to strengthen the credibility of empirical research in general and psychological science in particular.