A New Online Calibration Method for Multidimensional Computerized Adaptive Testing

Ping Chen; Chun Wang

doi:10.1007/s11336-015-9482-9

A New Online Calibration Method for Multidimensional Computerized Adaptive Testing

Ping Chen, Chun Wang

Psychology (Twin Cities)

Research output: Contribution to journal › Article › peer-review

24 Scopus citations

Abstract

Multidimensional-Method A (M-Method A) has been proposed as an efficient and effective online calibration method for multidimensional computerized adaptive testing (MCAT) (Chen & Xin, Paper presented at the 78th Meeting of the Psychometric Society, Arnhem, The Netherlands, 2013). However, a key assumption of M-Method A is that it treats person parameter estimates as their true values, thus this method might yield erroneous item calibration when person parameter estimates contain non-ignorable measurement errors. To improve the performance of M-Method A, this paper proposes a new MCAT online calibration method, namely, the full functional MLE-M-Method A (FFMLE-M-Method A). This new method combines the full functional MLE (Jones & Jin in Psychometrika 59:59–75, 1994; Stefanski & Carroll in Annals of Statistics 13:1335–1351, 1985) with the original M-Method A in an effort to correct for the estimation error of ability vector that might otherwise adversely affect the precision of item calibration. Two correction schemes are also proposed when implementing the new method. A simulation study was conducted to show that the new method generated more accurate item parameter estimation than the original M-Method A in almost all conditions.

Original language	English (US)
Pages (from-to)	674-701
Number of pages	28
Journal	Psychometrika
Volume	81
Issue number	3
DOIs	https://doi.org/10.1007/s11336-015-9482-9
State	Published - Sep 1 2016

Bibliographical note

Funding Information:
This study was partially supported by the National Natural Science Foundation of China (Grant No. 31300862), the Specialized Research Fund for the Doctoral Program of Higher Education (Grant No. 20130003120002), the Fundamental Research Funds for the Central Universities (Grant No. 2013YB26), the National Academy of Education/Spencer Fellowship (Grant No. 792269), and KLAS (Grant No. 130026509). Part of the paper was originally presented in 2014 annual meeting of the National Council on Measurement in Education, Philadelphia, Pennsylvania. The authors are indebted to the editor, associate editor, and four anonymous reviewers for their suggestions and comments on the earlier manuscript.

Publisher Copyright:
© 2015, The Psychometric Society.

Keywords

full functional maximum likelihood estimator
multidimensional computerized adaptive testing
multidimensional two-parameter logistic model
new item
online calibration
operational item

Publisher link

10.1007/s11336-015-9482-9

Find It

Find It at U of M Libraries

Cite this

@article{16113a7623ed45879a1cb03b7550de6f,

title = "A New Online Calibration Method for Multidimensional Computerized Adaptive Testing",

abstract = "Multidimensional-Method A (M-Method A) has been proposed as an efficient and effective online calibration method for multidimensional computerized adaptive testing (MCAT) (Chen & Xin, Paper presented at the 78th Meeting of the Psychometric Society, Arnhem, The Netherlands, 2013). However, a key assumption of M-Method A is that it treats person parameter estimates as their true values, thus this method might yield erroneous item calibration when person parameter estimates contain non-ignorable measurement errors. To improve the performance of M-Method A, this paper proposes a new MCAT online calibration method, namely, the full functional MLE-M-Method A (FFMLE-M-Method A). This new method combines the full functional MLE (Jones & Jin in Psychometrika 59:59–75, 1994; Stefanski & Carroll in Annals of Statistics 13:1335–1351, 1985) with the original M-Method A in an effort to correct for the estimation error of ability vector that might otherwise adversely affect the precision of item calibration. Two correction schemes are also proposed when implementing the new method. A simulation study was conducted to show that the new method generated more accurate item parameter estimation than the original M-Method A in almost all conditions.",

keywords = "full functional maximum likelihood estimator, multidimensional computerized adaptive testing, multidimensional two-parameter logistic model, new item, online calibration, operational item",

author = "Ping Chen and Chun Wang",

note = "Funding Information: This study was partially supported by the National Natural Science Foundation of China (Grant No. 31300862), the Specialized Research Fund for the Doctoral Program of Higher Education (Grant No. 20130003120002), the Fundamental Research Funds for the Central Universities (Grant No. 2013YB26), the National Academy of Education/Spencer Fellowship (Grant No. 792269), and KLAS (Grant No. 130026509). Part of the paper was originally presented in 2014 annual meeting of the National Council on Measurement in Education, Philadelphia, Pennsylvania. The authors are indebted to the editor, associate editor, and four anonymous reviewers for their suggestions and comments on the earlier manuscript. Publisher Copyright: {\textcopyright} 2015, The Psychometric Society.",

year = "2016",

month = sep,

day = "1",

doi = "10.1007/s11336-015-9482-9",

language = "English (US)",

volume = "81",

pages = "674--701",

journal = "Psychometrika",

issn = "0033-3123",

publisher = "Springer New York",

number = "3",

}

TY - JOUR

T1 - A New Online Calibration Method for Multidimensional Computerized Adaptive Testing

AU - Chen, Ping

AU - Wang, Chun

N1 - Funding Information: This study was partially supported by the National Natural Science Foundation of China (Grant No. 31300862), the Specialized Research Fund for the Doctoral Program of Higher Education (Grant No. 20130003120002), the Fundamental Research Funds for the Central Universities (Grant No. 2013YB26), the National Academy of Education/Spencer Fellowship (Grant No. 792269), and KLAS (Grant No. 130026509). Part of the paper was originally presented in 2014 annual meeting of the National Council on Measurement in Education, Philadelphia, Pennsylvania. The authors are indebted to the editor, associate editor, and four anonymous reviewers for their suggestions and comments on the earlier manuscript. Publisher Copyright: © 2015, The Psychometric Society.

PY - 2016/9/1

Y1 - 2016/9/1

N2 - Multidimensional-Method A (M-Method A) has been proposed as an efficient and effective online calibration method for multidimensional computerized adaptive testing (MCAT) (Chen & Xin, Paper presented at the 78th Meeting of the Psychometric Society, Arnhem, The Netherlands, 2013). However, a key assumption of M-Method A is that it treats person parameter estimates as their true values, thus this method might yield erroneous item calibration when person parameter estimates contain non-ignorable measurement errors. To improve the performance of M-Method A, this paper proposes a new MCAT online calibration method, namely, the full functional MLE-M-Method A (FFMLE-M-Method A). This new method combines the full functional MLE (Jones & Jin in Psychometrika 59:59–75, 1994; Stefanski & Carroll in Annals of Statistics 13:1335–1351, 1985) with the original M-Method A in an effort to correct for the estimation error of ability vector that might otherwise adversely affect the precision of item calibration. Two correction schemes are also proposed when implementing the new method. A simulation study was conducted to show that the new method generated more accurate item parameter estimation than the original M-Method A in almost all conditions.

AB - Multidimensional-Method A (M-Method A) has been proposed as an efficient and effective online calibration method for multidimensional computerized adaptive testing (MCAT) (Chen & Xin, Paper presented at the 78th Meeting of the Psychometric Society, Arnhem, The Netherlands, 2013). However, a key assumption of M-Method A is that it treats person parameter estimates as their true values, thus this method might yield erroneous item calibration when person parameter estimates contain non-ignorable measurement errors. To improve the performance of M-Method A, this paper proposes a new MCAT online calibration method, namely, the full functional MLE-M-Method A (FFMLE-M-Method A). This new method combines the full functional MLE (Jones & Jin in Psychometrika 59:59–75, 1994; Stefanski & Carroll in Annals of Statistics 13:1335–1351, 1985) with the original M-Method A in an effort to correct for the estimation error of ability vector that might otherwise adversely affect the precision of item calibration. Two correction schemes are also proposed when implementing the new method. A simulation study was conducted to show that the new method generated more accurate item parameter estimation than the original M-Method A in almost all conditions.

KW - full functional maximum likelihood estimator

KW - multidimensional computerized adaptive testing

KW - multidimensional two-parameter logistic model

KW - new item

KW - online calibration

KW - operational item

UR - http://www.scopus.com/inward/record.url?scp=84948141085&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84948141085&partnerID=8YFLogxK

U2 - 10.1007/s11336-015-9482-9

DO - 10.1007/s11336-015-9482-9

M3 - Article

C2 - 26608960

AN - SCOPUS:84948141085

SN - 0033-3123

VL - 81

SP - 674

EP - 701

JO - Psychometrika

JF - Psychometrika

IS - 3

ER -

A New Online Calibration Method for Multidimensional Computerized Adaptive Testing

Abstract

Bibliographical note

Keywords

Publisher link

Find It

Other files and links

Fingerprint

Cite this