Detecting Aberrant Behavior and Item Preknowledge

A Comparison of Mixture Modeling Method and Residual Method

Chun Wang, Gongjun Xu, Zhuoran Shang, Nathan R Kuncel

Research output: Contribution to journalArticle

5 Citations (Scopus)

Abstract

The modern web-based technology greatly popularizes computer-administered testing, also known as online testing. When these online tests are administered continuously within a certain “testing window,” many items are likely to be exposed and compromised, posing a type of test security concern. In addition, if the testing time is limited, another recognized aberrant behavior is rapid guessing, which refers to quickly answering an item without processing its meaning. Both cheating behavior and rapid guessing result in extremely short response times. This article introduces a mixture hierarchical item response theory model, using both response accuracy and response time information, to help differentiate aberrant behavior from normal behavior. The model-based approach is compared to the Bayesian residual-based fit statistic in both simulation study and two real data examples. Results show that the mixture model approach consistently outperforms the residual method in terms of correct detection rate and false positive error rate, in particular when the proportion of aberrance is high. Moreover, the model-based approach is also able to correctly identify compromised items better than residual method.

Original languageEnglish (US)
Pages (from-to)469-501
Number of pages33
JournalJournal of Educational and Behavioral Statistics
Volume43
Issue number4
DOIs
StatePublished - Aug 1 2018

Fingerprint

model theory
statistics
simulation
time

Keywords

  • aberrant behavior
  • item preknowledge
  • item response theory
  • mixture model
  • person-fit
  • response time

Cite this

Detecting Aberrant Behavior and Item Preknowledge : A Comparison of Mixture Modeling Method and Residual Method. / Wang, Chun; Xu, Gongjun; Shang, Zhuoran; Kuncel, Nathan R.

In: Journal of Educational and Behavioral Statistics, Vol. 43, No. 4, 01.08.2018, p. 469-501.

Research output: Contribution to journalArticle

@article{b245d59f6864447781d9cba615f1ee8d,
title = "Detecting Aberrant Behavior and Item Preknowledge: A Comparison of Mixture Modeling Method and Residual Method",
abstract = "The modern web-based technology greatly popularizes computer-administered testing, also known as online testing. When these online tests are administered continuously within a certain “testing window,” many items are likely to be exposed and compromised, posing a type of test security concern. In addition, if the testing time is limited, another recognized aberrant behavior is rapid guessing, which refers to quickly answering an item without processing its meaning. Both cheating behavior and rapid guessing result in extremely short response times. This article introduces a mixture hierarchical item response theory model, using both response accuracy and response time information, to help differentiate aberrant behavior from normal behavior. The model-based approach is compared to the Bayesian residual-based fit statistic in both simulation study and two real data examples. Results show that the mixture model approach consistently outperforms the residual method in terms of correct detection rate and false positive error rate, in particular when the proportion of aberrance is high. Moreover, the model-based approach is also able to correctly identify compromised items better than residual method.",
keywords = "aberrant behavior, item preknowledge, item response theory, mixture model, person-fit, response time",
author = "Chun Wang and Gongjun Xu and Zhuoran Shang and Kuncel, {Nathan R}",
year = "2018",
month = "8",
day = "1",
doi = "10.3102/1076998618767123",
language = "English (US)",
volume = "43",
pages = "469--501",
journal = "Journal of Educational and Behavioral Statistics",
issn = "1076-9986",
publisher = "SAGE Publications Inc.",
number = "4",

}

TY - JOUR

T1 - Detecting Aberrant Behavior and Item Preknowledge

T2 - A Comparison of Mixture Modeling Method and Residual Method

AU - Wang, Chun

AU - Xu, Gongjun

AU - Shang, Zhuoran

AU - Kuncel, Nathan R

PY - 2018/8/1

Y1 - 2018/8/1

N2 - The modern web-based technology greatly popularizes computer-administered testing, also known as online testing. When these online tests are administered continuously within a certain “testing window,” many items are likely to be exposed and compromised, posing a type of test security concern. In addition, if the testing time is limited, another recognized aberrant behavior is rapid guessing, which refers to quickly answering an item without processing its meaning. Both cheating behavior and rapid guessing result in extremely short response times. This article introduces a mixture hierarchical item response theory model, using both response accuracy and response time information, to help differentiate aberrant behavior from normal behavior. The model-based approach is compared to the Bayesian residual-based fit statistic in both simulation study and two real data examples. Results show that the mixture model approach consistently outperforms the residual method in terms of correct detection rate and false positive error rate, in particular when the proportion of aberrance is high. Moreover, the model-based approach is also able to correctly identify compromised items better than residual method.

AB - The modern web-based technology greatly popularizes computer-administered testing, also known as online testing. When these online tests are administered continuously within a certain “testing window,” many items are likely to be exposed and compromised, posing a type of test security concern. In addition, if the testing time is limited, another recognized aberrant behavior is rapid guessing, which refers to quickly answering an item without processing its meaning. Both cheating behavior and rapid guessing result in extremely short response times. This article introduces a mixture hierarchical item response theory model, using both response accuracy and response time information, to help differentiate aberrant behavior from normal behavior. The model-based approach is compared to the Bayesian residual-based fit statistic in both simulation study and two real data examples. Results show that the mixture model approach consistently outperforms the residual method in terms of correct detection rate and false positive error rate, in particular when the proportion of aberrance is high. Moreover, the model-based approach is also able to correctly identify compromised items better than residual method.

KW - aberrant behavior

KW - item preknowledge

KW - item response theory

KW - mixture model

KW - person-fit

KW - response time

UR - http://www.scopus.com/inward/record.url?scp=85045135275&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85045135275&partnerID=8YFLogxK

U2 - 10.3102/1076998618767123

DO - 10.3102/1076998618767123

M3 - Article

VL - 43

SP - 469

EP - 501

JO - Journal of Educational and Behavioral Statistics

JF - Journal of Educational and Behavioral Statistics

SN - 1076-9986

IS - 4

ER -