Limitations of true score variance to measure discriminating power

Psychometric simulation study

Seung Suk Kang, Angus MacDonald

Research output: Contribution to journalArticle

6 Citations (Scopus)

Abstract

Demonstrating a specific cognitive deficit usually involves comparing patients' performance on 2 or more tests. The psychometric confound occurs if the psychometric properties of these tests lead patients to show greater cognitive deficits in 1 domain. One way to avoid the psychometric confound is to use tests with a similar level of discriminating power, which is a test's ability to index true individual differences in classic psychometric theory. One suggested way to measure discriminating power is to calculate true score variance (L. J. Chapman & J. P. Chapman, 1978). Despite the centrality of these formulations, there is no systematic examination of the relationship between the observable property of true score variance and the latent property of discriminating power. The authors simulated administrations of free response tests and forced choice tests by creating different replicable ability scores for 2 groups, across a wide range of various psychometric properties (i.e., difficulty, reliability, observed variance, and number of items), and computing an ideal index of discriminating power. Simulation results indicated that true score variance had only limited ability to predict discriminating power (explained about 10% of variance in replicable ability scores). Furthermore, the ability varied across tests with wide ranges of psychometric variables, such as difficulty, observed variance, reliability, and number of items. Discriminating power depends on a complicated interaction of psychometric properties that is not well estimated solely by a test's true score variance.

Original languageEnglish (US)
Pages (from-to)300-306
Number of pages7
JournalJournal of Abnormal Psychology
Volume119
Issue number2
DOIs
StatePublished - Jan 1 2010

Fingerprint

Psychometrics
Aptitude
Power (Psychology)
Simulation
Individuality

Keywords

  • Differential deficit
  • Discriminating power
  • Generalized deficit
  • Psychometric confound
  • Simulation study

Cite this

Limitations of true score variance to measure discriminating power : Psychometric simulation study. / Kang, Seung Suk; MacDonald, Angus.

In: Journal of Abnormal Psychology, Vol. 119, No. 2, 01.01.2010, p. 300-306.

Research output: Contribution to journalArticle

@article{873b470468db44129a7e43960cfad3f8,
title = "Limitations of true score variance to measure discriminating power: Psychometric simulation study",
abstract = "Demonstrating a specific cognitive deficit usually involves comparing patients' performance on 2 or more tests. The psychometric confound occurs if the psychometric properties of these tests lead patients to show greater cognitive deficits in 1 domain. One way to avoid the psychometric confound is to use tests with a similar level of discriminating power, which is a test's ability to index true individual differences in classic psychometric theory. One suggested way to measure discriminating power is to calculate true score variance (L. J. Chapman & J. P. Chapman, 1978). Despite the centrality of these formulations, there is no systematic examination of the relationship between the observable property of true score variance and the latent property of discriminating power. The authors simulated administrations of free response tests and forced choice tests by creating different replicable ability scores for 2 groups, across a wide range of various psychometric properties (i.e., difficulty, reliability, observed variance, and number of items), and computing an ideal index of discriminating power. Simulation results indicated that true score variance had only limited ability to predict discriminating power (explained about 10{\%} of variance in replicable ability scores). Furthermore, the ability varied across tests with wide ranges of psychometric variables, such as difficulty, observed variance, reliability, and number of items. Discriminating power depends on a complicated interaction of psychometric properties that is not well estimated solely by a test's true score variance.",
keywords = "Differential deficit, Discriminating power, Generalized deficit, Psychometric confound, Simulation study",
author = "Kang, {Seung Suk} and Angus MacDonald",
year = "2010",
month = "1",
day = "1",
doi = "10.1037/a0018400",
language = "English (US)",
volume = "119",
pages = "300--306",
journal = "Journal of Abnormal Psychology",
issn = "0021-843X",
publisher = "American Psychological Association",
number = "2",

}

TY - JOUR

T1 - Limitations of true score variance to measure discriminating power

T2 - Psychometric simulation study

AU - Kang, Seung Suk

AU - MacDonald, Angus

PY - 2010/1/1

Y1 - 2010/1/1

N2 - Demonstrating a specific cognitive deficit usually involves comparing patients' performance on 2 or more tests. The psychometric confound occurs if the psychometric properties of these tests lead patients to show greater cognitive deficits in 1 domain. One way to avoid the psychometric confound is to use tests with a similar level of discriminating power, which is a test's ability to index true individual differences in classic psychometric theory. One suggested way to measure discriminating power is to calculate true score variance (L. J. Chapman & J. P. Chapman, 1978). Despite the centrality of these formulations, there is no systematic examination of the relationship between the observable property of true score variance and the latent property of discriminating power. The authors simulated administrations of free response tests and forced choice tests by creating different replicable ability scores for 2 groups, across a wide range of various psychometric properties (i.e., difficulty, reliability, observed variance, and number of items), and computing an ideal index of discriminating power. Simulation results indicated that true score variance had only limited ability to predict discriminating power (explained about 10% of variance in replicable ability scores). Furthermore, the ability varied across tests with wide ranges of psychometric variables, such as difficulty, observed variance, reliability, and number of items. Discriminating power depends on a complicated interaction of psychometric properties that is not well estimated solely by a test's true score variance.

AB - Demonstrating a specific cognitive deficit usually involves comparing patients' performance on 2 or more tests. The psychometric confound occurs if the psychometric properties of these tests lead patients to show greater cognitive deficits in 1 domain. One way to avoid the psychometric confound is to use tests with a similar level of discriminating power, which is a test's ability to index true individual differences in classic psychometric theory. One suggested way to measure discriminating power is to calculate true score variance (L. J. Chapman & J. P. Chapman, 1978). Despite the centrality of these formulations, there is no systematic examination of the relationship between the observable property of true score variance and the latent property of discriminating power. The authors simulated administrations of free response tests and forced choice tests by creating different replicable ability scores for 2 groups, across a wide range of various psychometric properties (i.e., difficulty, reliability, observed variance, and number of items), and computing an ideal index of discriminating power. Simulation results indicated that true score variance had only limited ability to predict discriminating power (explained about 10% of variance in replicable ability scores). Furthermore, the ability varied across tests with wide ranges of psychometric variables, such as difficulty, observed variance, reliability, and number of items. Discriminating power depends on a complicated interaction of psychometric properties that is not well estimated solely by a test's true score variance.

KW - Differential deficit

KW - Discriminating power

KW - Generalized deficit

KW - Psychometric confound

KW - Simulation study

UR - http://www.scopus.com/inward/record.url?scp=77952945257&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77952945257&partnerID=8YFLogxK

U2 - 10.1037/a0018400

DO - 10.1037/a0018400

M3 - Article

VL - 119

SP - 300

EP - 306

JO - Journal of Abnormal Psychology

JF - Journal of Abnormal Psychology

SN - 0021-843X

IS - 2

ER -