Estimating Between-Person and Within-Person Subscore Reliability with Profile Analysis

Okan Bulut, Mark L. Davison, Michael C. Rodriguez

Research output: Contribution to journalArticlepeer-review

22 Scopus citations

Abstract

Subscores are of increasing interest in educational and psychological testing due to their diagnostic function for evaluating examinees' strengths and weaknesses within particular domains of knowledge. Previous studies about the utility of subscores have mostly focused on the overall reliability of individual subscores and ignored the fact that subscores should be distinct and have added value over the total score. This study introduces a profile reliability approach that partitions the overall subscore reliability into within-person and between-person subscore reliability. The estimation of between-person reliability and within-person reliability coefficients is demonstrated using subscores from number-correct scoring, unidimensional and multidimensional item response theory scoring, and augmented scoring approaches via a simulation study and a real data study. The effects of various testing conditions, such as subtest length, correlations among subscores, and the number of subtests, are examined. Results indicate that there is a substantial trade-off between within-person and between-person reliability of subscores. Profile reliability coefficients can be useful in determining the extent to which subscores provide distinct and reliable information under various testing conditions.

Original languageEnglish (US)
Pages (from-to)86-104
Number of pages19
JournalMultivariate Behavioral Research
Volume52
Issue number1
DOIs
StatePublished - Jan 2 2017

Keywords

  • Subscore
  • multidimensional IRT
  • pattern
  • profile analysis
  • reliability

Fingerprint

Dive into the research topics of 'Estimating Between-Person and Within-Person Subscore Reliability with Profile Analysis'. Together they form a unique fingerprint.

Cite this