Effects of calibration sample size and item bank size on ability estimation in computerized adaptive testing

Alper Şahin, David J Weiss

Research output: Contribution to journalArticle

3 Citations (Scopus)

Abstract

This study aimed to investigate the effects of calibration sample size and item bank size on examinee ability estimation in computerized adaptive testing (CAT). For this purpose, a 500-item bank pre-calibrated using the three-parameter logistic model with 10,000 examinees was simulated. Calibration samples of varying sizes (150, 250, 350, 500, 750, 1,000, 2,000, 3,000, and 5,000) were selected from the parent sample, and item banks that represented small (100) and medium size (200 and 300) banks were drawn from the 500-item bank. Items in these banks were recalibrated using the drawn samples, and their estimated parameters were used in post-hoc simulations to re-estimate ability parameters for the simulated 10,000 examinees. The findings showed that ability estimates in CAT are robust against fluctuations in item parameter estimation and that accurate ability parameter estimates can be obtained with a calibration sample of 150 examinees. Moreover, a 200-item bank pre-calibrated with as few as 150 examinees can be used for some purposes in CAT as long as it has sufficient information at targeted ability levels.

Original languageEnglish (US)
Pages (from-to)1585-1595
Number of pages11
JournalKuram ve Uygulamada Egitim Bilimleri
Volume15
Issue number6
DOIs
StatePublished - Dec 1 2015

Fingerprint

bank
ability
fluctuation
parents
logistics
simulation

Keywords

  • Ability estimation accuracy
  • Calibration sample size
  • Computerized adaptive testing
  • Item response theory
  • Pretest item calibration

Cite this

Effects of calibration sample size and item bank size on ability estimation in computerized adaptive testing. / Şahin, Alper; Weiss, David J.

In: Kuram ve Uygulamada Egitim Bilimleri, Vol. 15, No. 6, 01.12.2015, p. 1585-1595.

Research output: Contribution to journalArticle

@article{e4548208b82a44a9b70d5c09e97b11b2,
title = "Effects of calibration sample size and item bank size on ability estimation in computerized adaptive testing",
abstract = "This study aimed to investigate the effects of calibration sample size and item bank size on examinee ability estimation in computerized adaptive testing (CAT). For this purpose, a 500-item bank pre-calibrated using the three-parameter logistic model with 10,000 examinees was simulated. Calibration samples of varying sizes (150, 250, 350, 500, 750, 1,000, 2,000, 3,000, and 5,000) were selected from the parent sample, and item banks that represented small (100) and medium size (200 and 300) banks were drawn from the 500-item bank. Items in these banks were recalibrated using the drawn samples, and their estimated parameters were used in post-hoc simulations to re-estimate ability parameters for the simulated 10,000 examinees. The findings showed that ability estimates in CAT are robust against fluctuations in item parameter estimation and that accurate ability parameter estimates can be obtained with a calibration sample of 150 examinees. Moreover, a 200-item bank pre-calibrated with as few as 150 examinees can be used for some purposes in CAT as long as it has sufficient information at targeted ability levels.",
keywords = "Ability estimation accuracy, Calibration sample size, Computerized adaptive testing, Item response theory, Pretest item calibration",
author = "Alper Şahin and Weiss, {David J}",
year = "2015",
month = "12",
day = "1",
doi = "10.12738/estp.2015.6.0102",
language = "English (US)",
volume = "15",
pages = "1585--1595",
journal = "Kuram ve Uygulamada Egitim Bilimleri",
issn = "1303-0485",
publisher = "Edam Egitim Danismanligi",
number = "6",

}

TY - JOUR

T1 - Effects of calibration sample size and item bank size on ability estimation in computerized adaptive testing

AU - Şahin, Alper

AU - Weiss, David J

PY - 2015/12/1

Y1 - 2015/12/1

N2 - This study aimed to investigate the effects of calibration sample size and item bank size on examinee ability estimation in computerized adaptive testing (CAT). For this purpose, a 500-item bank pre-calibrated using the three-parameter logistic model with 10,000 examinees was simulated. Calibration samples of varying sizes (150, 250, 350, 500, 750, 1,000, 2,000, 3,000, and 5,000) were selected from the parent sample, and item banks that represented small (100) and medium size (200 and 300) banks were drawn from the 500-item bank. Items in these banks were recalibrated using the drawn samples, and their estimated parameters were used in post-hoc simulations to re-estimate ability parameters for the simulated 10,000 examinees. The findings showed that ability estimates in CAT are robust against fluctuations in item parameter estimation and that accurate ability parameter estimates can be obtained with a calibration sample of 150 examinees. Moreover, a 200-item bank pre-calibrated with as few as 150 examinees can be used for some purposes in CAT as long as it has sufficient information at targeted ability levels.

AB - This study aimed to investigate the effects of calibration sample size and item bank size on examinee ability estimation in computerized adaptive testing (CAT). For this purpose, a 500-item bank pre-calibrated using the three-parameter logistic model with 10,000 examinees was simulated. Calibration samples of varying sizes (150, 250, 350, 500, 750, 1,000, 2,000, 3,000, and 5,000) were selected from the parent sample, and item banks that represented small (100) and medium size (200 and 300) banks were drawn from the 500-item bank. Items in these banks were recalibrated using the drawn samples, and their estimated parameters were used in post-hoc simulations to re-estimate ability parameters for the simulated 10,000 examinees. The findings showed that ability estimates in CAT are robust against fluctuations in item parameter estimation and that accurate ability parameter estimates can be obtained with a calibration sample of 150 examinees. Moreover, a 200-item bank pre-calibrated with as few as 150 examinees can be used for some purposes in CAT as long as it has sufficient information at targeted ability levels.

KW - Ability estimation accuracy

KW - Calibration sample size

KW - Computerized adaptive testing

KW - Item response theory

KW - Pretest item calibration

UR - http://www.scopus.com/inward/record.url?scp=84953375121&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84953375121&partnerID=8YFLogxK

U2 - 10.12738/estp.2015.6.0102

DO - 10.12738/estp.2015.6.0102

M3 - Article

VL - 15

SP - 1585

EP - 1595

JO - Kuram ve Uygulamada Egitim Bilimleri

JF - Kuram ve Uygulamada Egitim Bilimleri

SN - 1303-0485

IS - 6

ER -