Is Differential Noneffortful Responding Associated With Type I Error in Measurement Invariance Testing?

Research output: Contribution to journalArticlepeer-review

10 Scopus citations

Abstract

Low test-taking effort as a validity threat is common when examinees perceive an assessment context to have minimal personal value. Prior research has shown that in such contexts, subgroups may differ in their effort, which raises two concerns when making subgroup mean comparisons. First, it is unclear how differential effort could influence evaluations of scale property equivalence. Second, if attaining full scalar invariance, the degree to which differential effort can bias subgroup mean comparisons is unknown. To address these issues, a simulation study was conducted to examine the influence of differential noneffortful responding (NER) on evaluations of measurement invariance and latent mean comparisons. Results showed that as differential rates of NER grew, increased Type I errors of measurement invariance were observed only at the metric invariance level, while no negative effects were apparent for configural or scalar invariance. When full scalar invariance was correctly attained, differential NER led to bias of mean score comparisons as large as 0.18 standard deviations with a differential NER rate of 7%. These findings suggest that test users should evaluate and document potential differential NER prior to both conducting measurement quality analyses and reporting disaggregated subgroup mean performance.

Original languageEnglish (US)
Pages (from-to)957-979
Number of pages23
JournalEducational and Psychological Measurement
Volume81
Issue number5
DOIs
StatePublished - Oct 2021

Bibliographical note

Funding Information:
The author would like to thank Michael Rodriguez and Samuel Ihlenfeldt from the University of Minnesota for providing comments on an earlier draft of the manuscript. The author received no financial support for the research, authorship, and/or publication of this article.

Publisher Copyright:
© The Author(s) 2021.

Keywords

  • measurement invariance
  • noneffortful responding
  • subgroup comparisons
  • test-taking effort
  • validity

PubMed: MeSH publication types

  • Journal Article

Fingerprint

Dive into the research topics of 'Is Differential Noneffortful Responding Associated With Type I Error in Measurement Invariance Testing?'. Together they form a unique fingerprint.

Cite this