A Diagnostic Tree Model for Adaptive Assessment of Complex Cognitive Processes Using Multidimensional Response Options

Mark L. Davison, David J. Weiss, Joseph N. DeWeese, Ozge Ersan, Gina Biancarosa, Patrick C. Kennedy

Research output: Contribution to journalArticlepeer-review

1 Scopus citations

Abstract

A tree model for diagnostic educational testing is described along with Monte Carlo simulations designed to evaluate measurement accuracy based on the model. The model is implemented in an assessment of inferential reading comprehension, the Multiple-Choice Online Causal Comprehension Assessment (MOCCA), through a sequential, multidimensional, computerized adaptive testing (CAT) strategy. Assessment of the first dimension, reading comprehension (RC), is based on the three-parameter logistic model. For diagnostic and intervention purposes, the second dimension, called process propensity (PP), is used to classify struggling students based on their pattern of incorrect responses. In the simulation studies, CAT item selection rules and stopping rules were varied to evaluate their effect on measurement accuracy along dimension RC and classification accuracy along dimension PP. For dimension RC, methods that improved accuracy tended to increase test length. For dimension PP, however, item selection and stopping rules increased classification accuracy without materially increasing test length. A small live-testing pilot study confirmed some of the findings of the simulation studies. Development of the assessment has been guided by psychometric theory, Monte Carlo simulation results, and a theory of instruction and diagnosis.

Original languageEnglish (US)
Pages (from-to)914-941
Number of pages28
JournalJournal of Educational and Behavioral Statistics
Volume48
Issue number6
DOIs
StatePublished - Dec 2023

Bibliographical note

Publisher Copyright:
© 2023 AERA.

Keywords

  • classification testing
  • computerized adaptive testing
  • diagnostic testing
  • formative assessment
  • item response models
  • reading assessment

Fingerprint

Dive into the research topics of 'A Diagnostic Tree Model for Adaptive Assessment of Complex Cognitive Processes Using Multidimensional Response Options'. Together they form a unique fingerprint.

Cite this