TY - JOUR
T1 - A Diagnostic Tree Model for Adaptive Assessment of Complex Cognitive Processes Using Multidimensional Response Options
AU - Davison, Mark L.
AU - Weiss, David J.
AU - DeWeese, Joseph N.
AU - Ersan, Ozge
AU - Biancarosa, Gina
AU - Kennedy, Patrick C.
N1 - Publisher Copyright:
© 2023 AERA.
PY - 2023/12
Y1 - 2023/12
N2 - A tree model for diagnostic educational testing is described along with Monte Carlo simulations designed to evaluate measurement accuracy based on the model. The model is implemented in an assessment of inferential reading comprehension, the Multiple-Choice Online Causal Comprehension Assessment (MOCCA), through a sequential, multidimensional, computerized adaptive testing (CAT) strategy. Assessment of the first dimension, reading comprehension (RC), is based on the three-parameter logistic model. For diagnostic and intervention purposes, the second dimension, called process propensity (PP), is used to classify struggling students based on their pattern of incorrect responses. In the simulation studies, CAT item selection rules and stopping rules were varied to evaluate their effect on measurement accuracy along dimension RC and classification accuracy along dimension PP. For dimension RC, methods that improved accuracy tended to increase test length. For dimension PP, however, item selection and stopping rules increased classification accuracy without materially increasing test length. A small live-testing pilot study confirmed some of the findings of the simulation studies. Development of the assessment has been guided by psychometric theory, Monte Carlo simulation results, and a theory of instruction and diagnosis.
AB - A tree model for diagnostic educational testing is described along with Monte Carlo simulations designed to evaluate measurement accuracy based on the model. The model is implemented in an assessment of inferential reading comprehension, the Multiple-Choice Online Causal Comprehension Assessment (MOCCA), through a sequential, multidimensional, computerized adaptive testing (CAT) strategy. Assessment of the first dimension, reading comprehension (RC), is based on the three-parameter logistic model. For diagnostic and intervention purposes, the second dimension, called process propensity (PP), is used to classify struggling students based on their pattern of incorrect responses. In the simulation studies, CAT item selection rules and stopping rules were varied to evaluate their effect on measurement accuracy along dimension RC and classification accuracy along dimension PP. For dimension RC, methods that improved accuracy tended to increase test length. For dimension PP, however, item selection and stopping rules increased classification accuracy without materially increasing test length. A small live-testing pilot study confirmed some of the findings of the simulation studies. Development of the assessment has been guided by psychometric theory, Monte Carlo simulation results, and a theory of instruction and diagnosis.
KW - classification testing
KW - computerized adaptive testing
KW - diagnostic testing
KW - formative assessment
KW - item response models
KW - reading assessment
UR - http://www.scopus.com/inward/record.url?scp=85152297110&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85152297110&partnerID=8YFLogxK
U2 - 10.3102/10769986231158301
DO - 10.3102/10769986231158301
M3 - Article
AN - SCOPUS:85152297110
SN - 1076-9986
VL - 48
SP - 914
EP - 941
JO - Journal of Educational and Behavioral Statistics
JF - Journal of Educational and Behavioral Statistics
IS - 6
ER -