Item Pool Quality Control in Educational Testing: Change Point Model, Compound Risk, and Sequential Detection

Yunxiao Chen, Yi Hsuan Lee, Xiaoou Li

Research output: Contribution to journalArticlepeer-review

2 Scopus citations

Abstract

In standardized educational testing, test items are reused in multiple test administrations. To ensure the validity of test scores, the psychometric properties of items should remain unchanged over time. In this article, we consider the sequential monitoring of test items, in particular, the detection of abrupt changes to their psychometric properties, where a change can be caused by, for example, leakage of the item or change of the corresponding curriculum. We propose a statistical framework for the detection of abrupt changes in individual items. This framework consists of (1) a multistream Bayesian change point model describing sequential changes in items, (2) a compound risk function quantifying the risk in sequential decisions, and (3) sequential decision rules that control the compound risk. Throughout the sequential decision process, the proposed decision rule balances the trade-off between two sources of errors, the false detection of prechange items, and the nondetection of postchange items. An item-specific monitoring statistic is proposed based on an item response theory model that eliminates the confounding from the examinee population which changes over time. Sequential decision rules and their theoretical properties are developed under two settings: the oracle setting where the Bayesian change point model is completely known and a more realistic setting where some parameters of the model are unknown. Simulation studies are conducted under settings that mimic real operational tests.

Original languageEnglish (US)
Pages (from-to)322-352
Number of pages31
JournalJournal of Educational and Behavioral Statistics
Volume47
Issue number3
DOIs
StatePublished - Jun 2022

Bibliographical note

Publisher Copyright:
© 2021 The Author(s).

Keywords

  • compound decision
  • item preknowledge
  • multistream data
  • sequential change point detection
  • standardized testing
  • test security

Fingerprint

Dive into the research topics of 'Item Pool Quality Control in Educational Testing: Change Point Model, Compound Risk, and Sequential Detection'. Together they form a unique fingerprint.

Cite this