While learning analytics dashboards (LADs) are the most common form of LA intervention, there is limited evidence regarding their impact on students' learning outcomes. This systematic review synthesizes the findings of 38 research studies to investigate the impact of LADs on students' learning outcomes, encompassing achievement, participation, motivation, and attitudes. As we currently stand, there is no evidence to support the conclusion that LADs have lived up to the promise of improving academic achievement. Most studies reported negligible or small effects, with limited evidence from well-powered controlled experiments. Many studies merely compared users and non-users of LADs, confounding the dashboard effect with student engagement levels. Similarly, the impact of LADs on motivation and attitudes appeared modest, with only a few exceptions demonstrating significant effects. Small sample sizes in these studies highlight the need for larger-scale investigations to validate these findings. Notably, LADs showed a relatively substantial impact on student participation. Several studies reported medium to large effect sizes, suggesting that LADs can promote engagement and interaction in online learning environments. However, methodological shortcomings, such as reliance on traditional evaluation methods, self-selection bias, the assumption that access equates to usage, and a lack of standardized assessment tools, emerged as recurring issues. To advance the research line for LADs, researchers should use rigorous assessment methods and establish clear standards for evaluating learning constructs. Such efforts will advance our understanding of the potential of LADs to enhance learning outcomes and provide valuable insights for educators and researchers alike. 

Have Learning Analytics Dashboards Lived Up to the Hype? A Systematic Review of Impact on Students' Achievement, Motivation, Participation and Attitude

Developments in educational technology and learning analytics make it possible to automatically formulate and deploy personalized formative feedback to learners at scale. However, to be effective, the motivational and emotional impacts of such automated and personalized feedback need to be considered. The literature on feedback suggests that effective feedback, among other features, provides learners with a standard to compare their performance with, often called a reference frame. Past research has highlighted the emotional and motivational benefits of criterion‐referenced feedback (i.e., performance relative to a learning objective or mastery goal) compared to norm‐referenced feedback (performance relative to peers).Despite a substantial body of evidence regarding reference frame effects, important open questions remain. The questions encompass, for example, whether the benefits and drawbacks of norm‐referenced feedback apply in the same way to automated and personalize feedback messages and whether these effects apply to students uniformly. Further, the potential impacts of combining reference frames are largely unknown, even though combinations may be quite frequent in feedback practice. Finally, little research has been done on the effects of reference frames in computer‐supported collaborative learning, which differs from individual learning in meaningful ways. This study aims to contribute to addressing these open questions, thus providing insights into effective feedback design. Specifically, we aim to investigate usefulness perceptions as well as emotional and motivational effects of different reference frames—and their combination—in automated and personalized formative feedback on a computer‐supported collaborative learning task.A randomized field experiment with four feedback conditions (simple feedback, norm‐referenced, criterion‐referenced, and combined feedback) was conducted in a course within a teacher training program (N = 282). Collaborative groups worked on a learning task in the online learning environment, after which they received one of four possible automated and personalized formative feedback. We collected student data about feedback usefulness perceptions, motivational regulation, and achievement emotions to assess the differential effects of these feedback conditions.All feedback types were perceived as useful relative to the simple feedback condition. Norm‐referenced feedback showed detrimental effects for motivational regulation, whereas combined feedback led to more desirable motivational states. Further, criterion‐referenced feedback led to more positive emotions for overperformers and to more negative emotions for underperformers. The findings are discussed in light of the broader feedback literature, and recommendations for designing automated and personalized formative feedback messages for computer‐supported collaborative learning are presented.

Emotional and motivational effects of automated and personalized formative feedback: The role of reference frames

Mastery‐based assignments typically provide students with multiple opportunities to improve their performance, but getting students to take advantage of these opportunities is difficult. We report on the implementation of a two‐part series of nudges designed to improve students' engagement with and performance on mastery‐based assignments in high‐enrollment, introductory undergraduate courses. Students received one or two email‐based nudges for each assignment, providing both decision information and decision assistance. Delivered via email by a digital coaching system that provides tailored support to students, we evaluated the effectiveness of viewing versus not viewing these personalized nudges across introductory courses in mathematics, computer science and engineering over five terms at our large, research‐intensive university. We used multilevel modelling with a sample of 30,693 assignment scores nested within 79 assignments and 5349 undergraduate students to explore the relationship between students' viewing the nudges and their scores while controlling for performance in other courses and students' baseline proclivities to read email from the coaching system. We find that viewing versus not viewing nudges is significantly related to improved outcomes. Viewing the first of the two‐part nudge series is associated with a 3% increase in scores on average. Importantly, we found a significant interaction with prior academic performance such that students with the lowest average academic performance across all courses are predicted to benefit the most—up to 9%—from nudging. Positive impacts are observed in most courses where nudges were implemented and are found to be most impactful for later versus earlier assignments within a term. We discuss the implications of these results for supporting students' decision‐making within mastery‐based assignment schemes and suggest future research for tailored nudges to support student success.
What is already known about this topic

Nudges are known to be broadly effective, but heterogeneity is observed especially in education where the research base is relatively limited.
Mastery‐based grading is an alternative approach to grading that, among other features, emphasizes learning through an explicit feedback cycle of struggle and revision without penalty.
What this paper adds

This study assesses the effectiveness of personalized nudges in the context of mastery‐based grading in undergraduate STEM courses.
We examined the efficacy of viewing versus not viewing nudges in the real learning settings of three STEM courses enrolling more than 5000 students.
We found the nudges to be effective overall, correlated with improvements of approximately 3% in students' assignment scores and most impactful for students with the lowest average academic performance.
Implications for practice and/or policy

Nudging by email or text message has a low overhead cost and yet may provide materially significant gains in students' assessment scores.
Nudges can be effectively targeted to specific subsets of students and tailored based on students' prior and concurrent learning analytics data.



Viewing tailored nudges is correlated with improved mastery‐based assessment scores

Cognitive science of learning points to solutions for making use of existing study and instruction time more effectively and efficiently. However, solutions are not and cannot be one-size-fits-all. This paper outlines the danger of overreliance on specific strategies as one-size-fits-all recommendations and highlights instead the cognitive learning processes that facilitate meaningful and long-lasting learning. Three of the most commonly recommended strategies from cognitive science provide a starting point; understanding the underlying processes allows us to tailor these recommendations to implement at the right time, in the right way, for the right content, and for the right students. Recommendations regard teacher training, the funding and incentivizing of educational interventions, guidelines for the development of educational technologies, and policies that focus on using existing instructional time more wisely.

No Simple Solutions to Complex Problems: Cognitive Science Principles Can Guide but Not Prescribe Educational Decisions

This research investigates the effects of a success coaching program by examining the timing of the first appointment and the frequency of additional coaching. Using a fuzzy regression discontinuity approach, we provide Local Average Treatment Effects of the program when students make appointments early in the semester. Students with appointments scheduled before discontinuity had an average of two additional meetings compared to students who waited to make their first appointment. We find each coaching session yields an effect of about 0.2–0.3 grade points and approximately 0.8 additional credits earned. This article contributes to understanding of the timing and frequency effects for success coaching and recommends programs aim to have first interactions with students happen before the sixth week of classes and offer four to six appointments per semester for maximum impact. 

Student Success Coaching Is Best Delivered Early and Often

Surveying the (Virtual) Landscape: A scoping review of XR in postsecondary learning environments

It is difficult for instructors, and even students themselves, to become aware in real-time of inequitable behaviors occurring on student teams. Here, we explored a potential measure for inequitable teamwork drawing on data from a digital pedagogical tool designed to surface and disrupt such team behaviors. Students in a large, undergraduate business course completed seven surveys about team health (called team checks) at regular intervals throughout the term, providing information about team dynamics, contributions, and processes. The ways in which changes in students’ scores from team check to team check compared to the median changes for their team were used to identify the proportions of teams with outlier student scores. The results show that for every team size and team check item, the proportion of teams with outliers at the end of the term was smaller than at the beginning of the semester, indicating stabilization in how teammates evaluated their team experiences. In all but two cases, outlying students were not disproportionately likely to identify with historically marginalized groups based on gender or race/ethnicity. Thus, we did not broadly identify teamwork inequities in this specific context, but the method provides a basis for future studies about inequitable team behavior.

Teammates Stabilize over Time in How They Evaluate Their Team Experiences

Grade point average in "other" courses (GPAO) is an increasingly common measure used to control for prior academic performance and to predict future academic performance. In previous work, there are two distinct approaches to calculating GPAO, one based on only courses taken concurrently (term GPAO) and one based on all previous courses taken (cumulative GPAO). To our knowledge, no one has studied whether these methods for calculating the GPAO result in equivalent analyses and conclusions. As researchers often use one definition or the other without comment on why that choice was made, if the two calculations of GPAO are different, researchers might be inducing systematic error into their results and publishing potentially inaccurate conclusions. We looked at more than 3,700 courses at a public, research-intensive university over a decade and found limited evidence that the choice of GPAO calculation affects the conclusions. At most, one in seven courses could be affected. Further analysis suggests that there may be situations where one form of GPAO may be preferred over the other when it comes to examining inequity in courses or predicting student grades. However, we did not find sufficient evidence to universally recommend one form of GPAO over the other. 

How researchers calculate students’ grade point average in other courses has minimal impact

High-stakes exams play a large role in determining an introductory physics student's final grade. These exams have been shown to be inequitable, often to the detriment of students identifying with groups historically marginalized in physics. Given that exams are made up of individual questions, it is plausible that certain types of questions may be contributing to the observed equity gaps. In this paper, we examine whether that is the case for one type of forced-choice question, the complex multiple-choice (CMC) question. In a CMC question, students must select an answer choice that includes all correct responses and no incorrect responses from a list. To conduct our study, we used four years of data from Problem Roulette, an online program at our university that allows students to prepare for exams with actual questions from previous years' exams in a not-for-credit format. We categorized the 951 Physics II (Electricity and Magnetism) questions in the database as CMC or non-CMC. We found that students performed 8.0 percentage points worse on CMC questions than they did on non-CMC questions. In addition, we found differential impacts for low income students and domestic students relative to medium and high income students and international students. Regression models supported these descriptive findings. The results suggest that complex multiple-choice questions may be contributing to the equity gaps observed on physics exams. Considering, however, that CMC questions are more difficult for everyone, future research should examine the source of this difficulty and whether that source is functionally related to learning and assessment. For example, our data does not support using CMC questions instead of non-CMC as a way to differentiate top-performing students from everyone else.

Complex multiple-choice questions are inequitable for low-income and domestic students (and difficult for everyone)

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Caitlin Hayward

Author Tools

Chat about Author