Evaluating the Impact of Careless Responding on Aggregated-Scores: To Filter Unmotivated Examinees or Not?

Joseph A. Rios, Hongwen Guo, Liyang Mao, Ou Lydia Liu

Research output: Contribution to journalArticlepeer-review

25 Scopus citations

Abstract

When examinees' test-taking motivation is questionable, practitioners must determine whether careless responding is of practical concern and if so, decide on the best approach to filter such responses. As there has been insufficient research on these topics, the objectives of this study were to: a) evaluate the degree of underestimation in the true mean when careless responses are present, and b) compare the effectiveness of two filtering procedures in purifying biased aggregated-scores. Results demonstrated that: a) the true mean was underestimated by around 0.20 SDs if the total amount of careless responses exceeded 6.25%, 12.5%, and 12.5% for easy, moderately difficult, and difficult tests, respectively, and b) listwise deleting data from unmotivated examinees artificially inflated the true mean by as much as.42 SDs when ability was related to careless responding. Findings from this study have implications for when and how practitioners should handle careless responses for group-based low-stakes assessments.

Original languageEnglish (US)
Pages (from-to)74-104
Number of pages31
JournalInternational Journal of Testing
Volume17
Issue number1
DOIs
StatePublished - Jan 2 2017
Externally publishedYes

Keywords

  • careless responding
  • low-stakes testing
  • motivation
  • rapid guessing
  • test taking
  • validity

Fingerprint Dive into the research topics of 'Evaluating the Impact of Careless Responding on Aggregated-Scores: To Filter Unmotivated Examinees or Not?'. Together they form a unique fingerprint.

Cite this