Modeling individual differences in numerical reasoning speed as a random effect of response time limits

Robert Semmes; Mark L. Davison; Catherine Close

doi:10.1177/0146621611407305

Modeling individual differences in numerical reasoning speed as a random effect of response time limits

Robert Semmes, Mark L. Davison, Catherine Close

Educational Psychology

Research output: Contribution to journal › Article › peer-review

32 Scopus citations

Abstract

If numerical reasoning items are administered under time limits, will two dimensions be required to account for the responses, a numerical ability dimension and a speed dimension? A total of 182 college students answered 74 numerical reasoning items. Every item was taken with and without time limits by half the students. Three psychometric models were fit to the data-one including no time-limit effect, one including a fixed time-limit effect, and one including a random effect of time limits. The latter model best fit the data, suggesting that a speed dimension, the random effect of time limits, is needed to account for time-limited responses. The estimated reliability of the Speed scores was .39. Despite this low reliability, Speed scores were correlated with American College Testing (ACT) math scores and response times. Speed scores added significantly to the ACT math score variance accounted for by the numerical reasoning dimension in the model. A within-person log-odds ratio interpretation of the Speed score is proposed. Possible methods of improving Speed score reliability and methods for studying the speed dimension are discussed.

Original language	English (US)
Pages (from-to)	433-446
Number of pages	14
Journal	Applied Psychological Measurement
Volume	35
Issue number	6
DOIs	https://doi.org/10.1177/0146621611407305
State	Published - Sep 2011

Bibliographical note

Funding Information:
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by a United States Army Research Institute for the Behavioral and Social Sciences contract award number 1435-04-03-CT-74083 (Mark L. Davison) and by Grant No. R305C050059 from the Institute of Education Sciences in the U.S. Department of Education. The rights of research participants were protected and applicable human research guidelines were followed.

Keywords

Rasch model
hierarchical linear model
linear logistic model
multidimensional item-response theory
numerical reasoning
quantitative reasoning
random effects
response time limits
speededness
test time limits

Publisher link

10.1177/0146621611407305

Find It

Find It at U of M Libraries

Cite this

@article{a43abe3a5d8b4e0ea61aa8e681f339ac,

title = "Modeling individual differences in numerical reasoning speed as a random effect of response time limits",

abstract = "If numerical reasoning items are administered under time limits, will two dimensions be required to account for the responses, a numerical ability dimension and a speed dimension? A total of 182 college students answered 74 numerical reasoning items. Every item was taken with and without time limits by half the students. Three psychometric models were fit to the data-one including no time-limit effect, one including a fixed time-limit effect, and one including a random effect of time limits. The latter model best fit the data, suggesting that a speed dimension, the random effect of time limits, is needed to account for time-limited responses. The estimated reliability of the Speed scores was .39. Despite this low reliability, Speed scores were correlated with American College Testing (ACT) math scores and response times. Speed scores added significantly to the ACT math score variance accounted for by the numerical reasoning dimension in the model. A within-person log-odds ratio interpretation of the Speed score is proposed. Possible methods of improving Speed score reliability and methods for studying the speed dimension are discussed.",

keywords = "Rasch model, hierarchical linear model, linear logistic model, multidimensional item-response theory, numerical reasoning, quantitative reasoning, random effects, response time limits, speededness, test time limits",

author = "Robert Semmes and Davison, {Mark L.} and Catherine Close",

note = "Funding Information: The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by a United States Army Research Institute for the Behavioral and Social Sciences contract award number 1435-04-03-CT-74083 (Mark L. Davison) and by Grant No. R305C050059 from the Institute of Education Sciences in the U.S. Department of Education. The rights of research participants were protected and applicable human research guidelines were followed.",

year = "2011",

month = sep,

doi = "10.1177/0146621611407305",

language = "English (US)",

volume = "35",

pages = "433--446",

journal = "Applied Psychological Measurement",

issn = "0146-6216",

publisher = "Sage Publications Sage CA: Los Angeles, CA",

number = "6",

}

TY - JOUR

T1 - Modeling individual differences in numerical reasoning speed as a random effect of response time limits

AU - Semmes, Robert

AU - Davison, Mark L.

AU - Close, Catherine

N1 - Funding Information: The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by a United States Army Research Institute for the Behavioral and Social Sciences contract award number 1435-04-03-CT-74083 (Mark L. Davison) and by Grant No. R305C050059 from the Institute of Education Sciences in the U.S. Department of Education. The rights of research participants were protected and applicable human research guidelines were followed.

PY - 2011/9

Y1 - 2011/9

N2 - If numerical reasoning items are administered under time limits, will two dimensions be required to account for the responses, a numerical ability dimension and a speed dimension? A total of 182 college students answered 74 numerical reasoning items. Every item was taken with and without time limits by half the students. Three psychometric models were fit to the data-one including no time-limit effect, one including a fixed time-limit effect, and one including a random effect of time limits. The latter model best fit the data, suggesting that a speed dimension, the random effect of time limits, is needed to account for time-limited responses. The estimated reliability of the Speed scores was .39. Despite this low reliability, Speed scores were correlated with American College Testing (ACT) math scores and response times. Speed scores added significantly to the ACT math score variance accounted for by the numerical reasoning dimension in the model. A within-person log-odds ratio interpretation of the Speed score is proposed. Possible methods of improving Speed score reliability and methods for studying the speed dimension are discussed.

AB - If numerical reasoning items are administered under time limits, will two dimensions be required to account for the responses, a numerical ability dimension and a speed dimension? A total of 182 college students answered 74 numerical reasoning items. Every item was taken with and without time limits by half the students. Three psychometric models were fit to the data-one including no time-limit effect, one including a fixed time-limit effect, and one including a random effect of time limits. The latter model best fit the data, suggesting that a speed dimension, the random effect of time limits, is needed to account for time-limited responses. The estimated reliability of the Speed scores was .39. Despite this low reliability, Speed scores were correlated with American College Testing (ACT) math scores and response times. Speed scores added significantly to the ACT math score variance accounted for by the numerical reasoning dimension in the model. A within-person log-odds ratio interpretation of the Speed score is proposed. Possible methods of improving Speed score reliability and methods for studying the speed dimension are discussed.

KW - Rasch model

KW - hierarchical linear model

KW - linear logistic model

KW - multidimensional item-response theory

KW - numerical reasoning

KW - quantitative reasoning

KW - random effects

KW - response time limits

KW - speededness

KW - test time limits

UR - http://www.scopus.com/inward/record.url?scp=79961040574&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=79961040574&partnerID=8YFLogxK

U2 - 10.1177/0146621611407305

DO - 10.1177/0146621611407305

M3 - Article

AN - SCOPUS:79961040574

SN - 0146-6216

VL - 35

SP - 433

EP - 446

JO - Applied Psychological Measurement

JF - Applied Psychological Measurement

IS - 6

ER -

Modeling individual differences in numerical reasoning speed as a random effect of response time limits

Abstract

Bibliographical note

Keywords

Publisher link

Find It

Other files and links

Fingerprint

Cite this