Choosing between CDM and Unidimensional IRT: The Proportional Reasoning Test Case

Cognitive diagnosis models (CDMs) have gained increasing popularity recently because of their potential to provide diagnostic inferences that can inform learning and teaching. However, the development of cognitive diagnostic assessments (CDAs) is lagging behind the development of the associated psychometric models. At present, it is not clear whether and how much extra information can be obtained by analyzing CDA data with a CDM rather than simply using unidimensional item response theory models. This study fits CDMs and three-parameter logistic (3PL) model to the data from a CDA. Results showed that the CDMs fitted the data well, but the 3PL model did not. Compared with the 3PL model, the CDMs can highlight student differences from a multidimensional perspective, thus offering finer-grained information that unidimensional IRT models are not capable of providing.