Constructing Subscores That Add Validity: A Case Study of Identifying Students at Risk

Gina Biancarosa, Patrick C. Kennedy, Sarah E. Carlson, Hyeon Jin Yoon, Ben Seipel, Bowen Liu, Mark L Davison

Research output: Contribution to journalArticle

Abstract

Prior research suggests that subscores from a single achievement test seldom add value over a single total score. Such scores typically correspond to subcontent areas in the total content domain, but content subdomains might not provide a sound basis for subscores. Using scores on an inferential reading comprehension test from 625 third, fourth, and fifth graders, two new methods of creating subscores were explored. Three subscores were based on the types of incorrect answers given by students. The fourth was based on temporal efficiency in giving correct answers. All four scores were reliable. The three subscores based on incorrect answers added value and validity. In logistic regression analyses predicting failure to reach proficiency on a statewide test, models including subscores fit better than the model with a single total score. Including the pattern of incorrect responses improved fit in all three grades, whereas including the comprehension efficiency score only modestly improved fit in fourth and fifth grades, but not third grade. Area under the curve (AUC) statistics from receiver operating characteristic (ROC) curves based on the various models were higher for models including subscores than those without subscores. Implications for using models with and without subscores are illustrated and discussed.

Original languageEnglish (US)
Pages (from-to)65-84
Number of pages20
JournalEducational and Psychological Measurement
Volume79
Issue number1
DOIs
StatePublished - Feb 1 2019

Fingerprint

Students
Efficiency
ROC Curve
Area Under Curve
Reading
student
Logistic Models
Regression Analysis
comprehension
Research
efficiency
Model
achievement test
Receiver Operating Characteristic Curve
Logistics
Logistic Regression
value added
Statistics
Acoustic waves
recipient

Keywords

  • at-risk screening
  • diagnostic testing
  • formative assessment
  • reading comprehension
  • subscores

Cite this

Constructing Subscores That Add Validity : A Case Study of Identifying Students at Risk. / Biancarosa, Gina; Kennedy, Patrick C.; Carlson, Sarah E.; Yoon, Hyeon Jin; Seipel, Ben; Liu, Bowen; Davison, Mark L.

In: Educational and Psychological Measurement, Vol. 79, No. 1, 01.02.2019, p. 65-84.

Research output: Contribution to journalArticle

Biancarosa, Gina ; Kennedy, Patrick C. ; Carlson, Sarah E. ; Yoon, Hyeon Jin ; Seipel, Ben ; Liu, Bowen ; Davison, Mark L. / Constructing Subscores That Add Validity : A Case Study of Identifying Students at Risk. In: Educational and Psychological Measurement. 2019 ; Vol. 79, No. 1. pp. 65-84.
@article{98efdc5dcf8d44d6bee3f5bcb5115006,
title = "Constructing Subscores That Add Validity: A Case Study of Identifying Students at Risk",
abstract = "Prior research suggests that subscores from a single achievement test seldom add value over a single total score. Such scores typically correspond to subcontent areas in the total content domain, but content subdomains might not provide a sound basis for subscores. Using scores on an inferential reading comprehension test from 625 third, fourth, and fifth graders, two new methods of creating subscores were explored. Three subscores were based on the types of incorrect answers given by students. The fourth was based on temporal efficiency in giving correct answers. All four scores were reliable. The three subscores based on incorrect answers added value and validity. In logistic regression analyses predicting failure to reach proficiency on a statewide test, models including subscores fit better than the model with a single total score. Including the pattern of incorrect responses improved fit in all three grades, whereas including the comprehension efficiency score only modestly improved fit in fourth and fifth grades, but not third grade. Area under the curve (AUC) statistics from receiver operating characteristic (ROC) curves based on the various models were higher for models including subscores than those without subscores. Implications for using models with and without subscores are illustrated and discussed.",
keywords = "at-risk screening, diagnostic testing, formative assessment, reading comprehension, subscores",
author = "Gina Biancarosa and Kennedy, {Patrick C.} and Carlson, {Sarah E.} and Yoon, {Hyeon Jin} and Ben Seipel and Bowen Liu and Davison, {Mark L}",
year = "2019",
month = "2",
day = "1",
doi = "10.1177/0013164418763255",
language = "English (US)",
volume = "79",
pages = "65--84",
journal = "Educational and Psychological Measurement",
issn = "0013-1644",
publisher = "SAGE Publications Inc.",
number = "1",

}

TY - JOUR

T1 - Constructing Subscores That Add Validity

T2 - A Case Study of Identifying Students at Risk

AU - Biancarosa, Gina

AU - Kennedy, Patrick C.

AU - Carlson, Sarah E.

AU - Yoon, Hyeon Jin

AU - Seipel, Ben

AU - Liu, Bowen

AU - Davison, Mark L

PY - 2019/2/1

Y1 - 2019/2/1

N2 - Prior research suggests that subscores from a single achievement test seldom add value over a single total score. Such scores typically correspond to subcontent areas in the total content domain, but content subdomains might not provide a sound basis for subscores. Using scores on an inferential reading comprehension test from 625 third, fourth, and fifth graders, two new methods of creating subscores were explored. Three subscores were based on the types of incorrect answers given by students. The fourth was based on temporal efficiency in giving correct answers. All four scores were reliable. The three subscores based on incorrect answers added value and validity. In logistic regression analyses predicting failure to reach proficiency on a statewide test, models including subscores fit better than the model with a single total score. Including the pattern of incorrect responses improved fit in all three grades, whereas including the comprehension efficiency score only modestly improved fit in fourth and fifth grades, but not third grade. Area under the curve (AUC) statistics from receiver operating characteristic (ROC) curves based on the various models were higher for models including subscores than those without subscores. Implications for using models with and without subscores are illustrated and discussed.

AB - Prior research suggests that subscores from a single achievement test seldom add value over a single total score. Such scores typically correspond to subcontent areas in the total content domain, but content subdomains might not provide a sound basis for subscores. Using scores on an inferential reading comprehension test from 625 third, fourth, and fifth graders, two new methods of creating subscores were explored. Three subscores were based on the types of incorrect answers given by students. The fourth was based on temporal efficiency in giving correct answers. All four scores were reliable. The three subscores based on incorrect answers added value and validity. In logistic regression analyses predicting failure to reach proficiency on a statewide test, models including subscores fit better than the model with a single total score. Including the pattern of incorrect responses improved fit in all three grades, whereas including the comprehension efficiency score only modestly improved fit in fourth and fifth grades, but not third grade. Area under the curve (AUC) statistics from receiver operating characteristic (ROC) curves based on the various models were higher for models including subscores than those without subscores. Implications for using models with and without subscores are illustrated and discussed.

KW - at-risk screening

KW - diagnostic testing

KW - formative assessment

KW - reading comprehension

KW - subscores

UR - http://www.scopus.com/inward/record.url?scp=85059046750&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85059046750&partnerID=8YFLogxK

U2 - 10.1177/0013164418763255

DO - 10.1177/0013164418763255

M3 - Article

VL - 79

SP - 65

EP - 84

JO - Educational and Psychological Measurement

JF - Educational and Psychological Measurement

SN - 0013-1644

IS - 1

ER -