The past 10 years have seen rapid growth of online (web-based) data collection across the behavioural sciences. Despite the many important contributions of such studies, some researchers have concerns about the reduction in experimental control when research moves outside of laboratory conditions. This paper provides an accessible overview of the issues that can adversely affect data quality in online experiments, with particular focus on cognitive studies of memory and language. I provide checklists for researchers setting up such experiments to help improve data quality. These recommendations focus on three key aspects of experimental design: the technology choices made by researchers and participants, participant recruitment methods, and the performance of participants during experiments. I argue that ensuring high data quality for online experiments requires significant effort prior to data collection to maintain the credibility of our rapidly expanding evidence base. With such safeguards in place, online experiments will continue to provide important, paradigm-changing opportunities across the behavioural sciences. 

Moving experimental psychology online: How to obtain high quality data when we can’t see our participants

Music commonly appears in behavioral contexts in which it can be seen as playing a functional role, as when a parent sings a lullaby with the goal of soothing a baby. Humans readily make inferences, based on the sounds they hear, regarding the behavioral contexts associated with music. These inferences tend to be accurate, even if the songs are in foreign languages or unfamiliar musical idioms; upon hearing a Blackfoot lullaby, a Korean listener with no experience of Blackfoot music, language, or broader culture is far more likely to judge the music's function as "used to soothe a baby" than "used for dancing". Are such inferences shaped by musical exposure or does the human mind naturally detect links between musical form and function of these kinds? Children's developing experience of music provides a clear test of this question. We studied musical inferences in a large sample of children recruited online (N = 5,033), who heard dance, lullaby, and healing songs from 70 world cultures and who were tasked with guessing the original behavioral context in which each was performed. Children reliably inferred the original behavioral contexts with only minimal improvement in performance from the youngest (age 4) to the oldest (age 16), providing little evidence for an effect of experience. Children's inferences tightly correlated with those of adults for the same songs, as collected from a similar online experiment (N = 98,150). Moreover, similar acoustical features were predictive of the inferences of both samples. These findings suggest that accurate inferences about the behavioral contexts of music, driven by universal links between form and function in music across cultures, do not always require extensive musical experience. (PsycInfo Database Record (c) 2023 APA, all rights reserved). 

/pdf/children-infer-the-behavioral-contexts-of-unfamiliar-foreign-2c3snkgl.pdf

Children infer the behavioral contexts of unfamiliar foreign songs.

The dominant paradigm of experiments in the social and behavioral sciences views an experiment as a test of a theory, where the theory is assumed to generalize beyond the experiment’s specific conditions. According to this view, which Alan Newell once characterized as “playing twenty questions with nature,” theory is advanced one experiment at a time, and the integration of disparate findings is assumed to happen via the scientific publishing process. In this article, we argue that the process of integration is at best inefficient, and at worst it does not, in fact, occur. We further show that the challenge of integration cannot be adequately addressed by recently proposed reforms that focus on the reliability and replicability of individual findings, nor simply by conducting more or larger experiments. Rather, the problem arises from the imprecise nature of social and behavioral theories and, consequently, a lack of commensurability across experiments conducted under different conditions. Therefore, researchers must fundamentally rethink how they design experiments and how the experiments relate to theory. We specifically describe an alternative framework, integrative experiment design, which intrinsically promotes commensurability and continuous integration of knowledge. In this paradigm, researchers explicitly map the design space of possible experiments associated with a given research question, embracing many potentially relevant theories rather than focusing on just one. The researchers then iteratively generate theories and test them with experiments explicitly sampled from the design space, allowing results to be integrated across experiments. Given recent methodological and technological developments, we conclude that this approach is feasible and would generate more-reliable, more-cumulative empirical and theoretical knowledge than the current paradigm—and with far greater efficiency. 

Beyond Playing 20 Questions with Nature: Integrative Experiment Design in the Social and Behavioral Sciences

First steps into the pupillometry multiverse of developmental science.

The past 10 years have seen rapid growth of online (web-based) data collection across the behavioural sciences. Many researchers have well-founded concerns about the lack of experimental control that arises when research moves outside of laboratory conditions. This paper provides a broad perspective on how to safeguard data quality in online experiments. I caution researchers against viewing online experiments as being necessarily cheaper, easier or more efficient than in-person testing. Ensuring appropriate data quality for online experiments requires significant effort prior to data collection to maintain the credibility of our rapidly expanding evidence base. I present a set of recommendations for researchers based on three key aspects of online data collection: technology, recruitment, and participant performance. With such safeguards in place online experiments will continue to provide important, paradigm-changing opportunities. 

Moving Experimental Psychology Online: How to Maintain Data Quality When We Can’t See Our Participants

Widespread failures of replication and generalization are, ironically, a scientific triumph, in that they confirm the fundamental metascientific theory that underlies our field. Generalizable and replicable findings require testing large numbers of subjects from a wide range of demographics with a large, randomly-sampled stimulus set, and using a variety of experimental parameters. Because few studies accomplish any of this, meta-scientists predict that findings will frequently fail to replicate or generalize. We argue that to be more robust and replicable, developmental psychology needs to find a mechanism for collecting data at a greater scale and from more diverse populations. Luckily, this mechanism already exists as follows: Citizen science, in which large numbers of uncompensated volunteers provide data. While best-known for its contributions to astronomy and ecology, citizen science has also produced major findings in neuroscience and psychology, and increasingly in developmental psychology. We provide examples, address practical challenges, discuss limitations, and compare to other methods of obtaining large datasets. Ultimately, we argue that the range of pose challenge Pashler Harris, 2012; Stanley 2018; Yarkoni, 2019). If we cannot trust the data we collect or the inferences we draw, what is the point of doing research?

Developmental psychologists should adopt citizen science to improve generalization and reproducibility

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Wei Li

Author Tools

Chat about Author

Papers

Developmental psychologists should adopt citizen science to improve generalization and reproducibility