ManyClasses 1: Assessing the Generalizable Effect of Immediate Feedback Versus Delayed Feedback Across Many College Classes

Emily R. Fyfe, Joshua R. de Leeuw, Paulo F. Carvalho, Robert L. Goldstone, Janelle Sherman, David Admiraal, Laura K. Alford, Alison Bonner, Chad E. Brassil, Christopher A. Brooks, Tracey Carbonetto, Sau Hou Chang, Laura Cruz, Melina Czymoniewicz-Klippel, Frances Daniel, Michelle Driessen, Noel Habashy, Carrie L. Hanson-Bradley, Edward R. Hirt, Virginia Hojas CarbonellDaniel K. Jackson, Shay Jones, Jennifer L. Keagy, Brandi Keith, Sarah J. Malmquist, Barry McQuarrie, Kelsey J. Metzger, Maung K. Min, Sameer Patil, Ryan S. Patrick, Etienne Pelaprat, Maureen L. Petrunich-Rutherford, Meghan R. Porter, Kristina Prescott, Cathrine Reck, Terri Renner, Eric Robbins, Adam R. Smith, Phil Stuczynski, Jaye Thompson, Nikolaos Tsotakos, Judith K. Turk, Kyle Unruh, Jennifer D. Webb, Stephanie N. Whitehead, Elaine C. Wisniewski, Ke Anne Zhang, Benjamin A. Motz

Research output: Contribution to journalArticlepeer-review

9 Scopus citations

Abstract

Psychology researchers have long attempted to identify educational practices that improve student learning. However, experimental research on these practices is often conducted in laboratory contexts or in a single course, which threatens the external validity of the results. In this article, we establish an experimental paradigm for evaluating the benefits of recommended practices across a variety of authentic educational contexts—a model we call ManyClasses. The core feature is that researchers examine the same research question and measure the same experimental effect across many classes spanning a range of topics, institutions, teacher implementations, and student populations. We report the first ManyClasses study, in which we examined how the timing of feedback on class assignments, either immediate or delayed by a few days, affected subsequent performance on class assessments. Across 38 classes, the overall estimate for the effect of feedback timing was 0.002 (95% highest density interval = [−0.05, 0.05]), which indicates that there was no effect of immediate feedback compared with delayed feedback on student learning that generalizes across classes. Furthermore, there were no credibly nonzero effects for 40 preregistered moderators related to class-level and student-level characteristics. Yet our results provide hints that in certain kinds of classes, which were undersampled in the current study, there may be modest advantages for delayed feedback. More broadly, these findings provide insights regarding the feasibility of conducting within-class randomized experiments across a range of naturally occurring learning environments.

Original languageEnglish (US)
JournalAdvances in Methods and Practices in Psychological Science
Volume4
Issue number3
DOIs
StatePublished - Jul 1 2021

Bibliographical note

Funding Information:
We thank John K. Kruschke for feedback on the analysis plan and Andrew C. Butler and two anonymous reviewers for their comments on an earlier draft of this article. We also acknowledge and thank the many people who assisted with this study, including Aaron Neal (Unizin), Jill Buban (Unizin), Stephan Nicklow (Unizin), Kara Armstrong (Unizin), David Goodrum (Oregon State), Sol Bermann (University of Michigan [UMich]), Sean DeMonner (UMich), Paul Robinson (UMich), Kelly Cruz (UMich), James Hilton (UMich), Matthew Kaplan (UMich), Lisa Emery (UMich), Angela Linse (Penn State University), Stacy Morrone (Indiana University [IU]), Erik Scull (IU), Greg Siering (IU), John Gosney (IU), Andrew Korty (IU), Andrew Nill (IU), Juliet Aders (IU), Katie Morris (IU), Jeffrey Goetz (IU), LeAnna Faubion (IU), Ryan Ballard (IU), Bethany Johnson (IU), Emily Oakes (IU), Sara Chambers (IU), Julie Lorah (IU), Dubravka Svetina (IU), Amy Goodburn (University of Nebraska Lincoln [UNL]), Heath Tuttle (UNL), Matt Morton (UNL), Sydney Brown (UNL), Tammie Herrington (UNL), Donalee Attardo (University of Minnesota [UMN]), Robert Alberti (UMN), Karen Hanson (UMN), Emily Ronning (UMN), Lauren Marsh (UMN), Paul Savereide (UMN), and Brian Dahlin (UMN). This study was supported with supplemental funding from the Department of Psychological and Brain Sciences and University Information Technology Services?s division of Learning Technologies at Indiana University Bloomington.

Publisher Copyright:
© The Author(s) 2021.

Keywords

  • education
  • evidence-based practices
  • experiment
  • feedback
  • reproducibility

Fingerprint

Dive into the research topics of 'ManyClasses 1: Assessing the Generalizable Effect of Immediate Feedback Versus Delayed Feedback Across Many College Classes'. Together they form a unique fingerprint.

Cite this