Multivariate Hypothesis Testing Methods for Evaluating Significant Individual Change

Chun Wang, David J Weiss

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

The measurement of individual change has been an important topic in both education and psychology. For instance, teachers are interested in whether students have significantly improved (e.g., learned) from instruction, and counselors are interested in whether particular behaviors have been significantly changed after certain interventions. Although classical test methods have been unable to adequately resolve the problems in measuring change, recent approaches for measuring change have begun to use item response theory (IRT). However, all prior methods mainly focus on testing whether growth is significant at the group level. The present research targets a key research question: Is the “change” in latent trait estimates for each individual significant across occasions? Many researchers have addressed this research question assuming that the latent trait is unidimensional. This research generalizes their earlier work and proposes four hypothesis testing methods to evaluate individual change on multiple latent traits: a multivariate Z-test, a multivariate likelihood ratio test, a multivariate score test, and a Kullback–Leibler test. Simulation results show that these tests hold promise of detecting individual change with low Type I error and high power. A real-data example from an educational assessment illustrates the application of the proposed methods.

Original languageEnglish (US)
Pages (from-to)221-239
Number of pages19
JournalApplied Psychological Measurement
Volume42
Issue number3
DOIs
StatePublished - May 1 2018

Fingerprint

hypothesis testing
Research
Educational Measurement
Research Personnel
Students
Psychology
counselor
Education
psychology
Growth
instruction
simulation
teacher
education
Group
student

Keywords

  • Kullback–Leibler test
  • individual change
  • likelihood ratio test
  • multidimensional item response theory
  • score test

PubMed: MeSH publication types

  • Journal Article
  • Review

Cite this

Multivariate Hypothesis Testing Methods for Evaluating Significant Individual Change. / Wang, Chun; Weiss, David J.

In: Applied Psychological Measurement, Vol. 42, No. 3, 01.05.2018, p. 221-239.

Research output: Contribution to journalArticle

@article{5243227203c245b3a8e3d7aeeebec1af,
title = "Multivariate Hypothesis Testing Methods for Evaluating Significant Individual Change",
abstract = "The measurement of individual change has been an important topic in both education and psychology. For instance, teachers are interested in whether students have significantly improved (e.g., learned) from instruction, and counselors are interested in whether particular behaviors have been significantly changed after certain interventions. Although classical test methods have been unable to adequately resolve the problems in measuring change, recent approaches for measuring change have begun to use item response theory (IRT). However, all prior methods mainly focus on testing whether growth is significant at the group level. The present research targets a key research question: Is the “change” in latent trait estimates for each individual significant across occasions? Many researchers have addressed this research question assuming that the latent trait is unidimensional. This research generalizes their earlier work and proposes four hypothesis testing methods to evaluate individual change on multiple latent traits: a multivariate Z-test, a multivariate likelihood ratio test, a multivariate score test, and a Kullback–Leibler test. Simulation results show that these tests hold promise of detecting individual change with low Type I error and high power. A real-data example from an educational assessment illustrates the application of the proposed methods.",
keywords = "Kullback–Leibler test, individual change, likelihood ratio test, multidimensional item response theory, score test",
author = "Chun Wang and Weiss, {David J}",
year = "2018",
month = "5",
day = "1",
doi = "10.1177/0146621617726787",
language = "English (US)",
volume = "42",
pages = "221--239",
journal = "Applied Psychological Measurement",
issn = "0146-6216",
publisher = "Sage Publications Sage CA: Los Angeles, CA",
number = "3",

}

TY - JOUR

T1 - Multivariate Hypothesis Testing Methods for Evaluating Significant Individual Change

AU - Wang, Chun

AU - Weiss, David J

PY - 2018/5/1

Y1 - 2018/5/1

N2 - The measurement of individual change has been an important topic in both education and psychology. For instance, teachers are interested in whether students have significantly improved (e.g., learned) from instruction, and counselors are interested in whether particular behaviors have been significantly changed after certain interventions. Although classical test methods have been unable to adequately resolve the problems in measuring change, recent approaches for measuring change have begun to use item response theory (IRT). However, all prior methods mainly focus on testing whether growth is significant at the group level. The present research targets a key research question: Is the “change” in latent trait estimates for each individual significant across occasions? Many researchers have addressed this research question assuming that the latent trait is unidimensional. This research generalizes their earlier work and proposes four hypothesis testing methods to evaluate individual change on multiple latent traits: a multivariate Z-test, a multivariate likelihood ratio test, a multivariate score test, and a Kullback–Leibler test. Simulation results show that these tests hold promise of detecting individual change with low Type I error and high power. A real-data example from an educational assessment illustrates the application of the proposed methods.

AB - The measurement of individual change has been an important topic in both education and psychology. For instance, teachers are interested in whether students have significantly improved (e.g., learned) from instruction, and counselors are interested in whether particular behaviors have been significantly changed after certain interventions. Although classical test methods have been unable to adequately resolve the problems in measuring change, recent approaches for measuring change have begun to use item response theory (IRT). However, all prior methods mainly focus on testing whether growth is significant at the group level. The present research targets a key research question: Is the “change” in latent trait estimates for each individual significant across occasions? Many researchers have addressed this research question assuming that the latent trait is unidimensional. This research generalizes their earlier work and proposes four hypothesis testing methods to evaluate individual change on multiple latent traits: a multivariate Z-test, a multivariate likelihood ratio test, a multivariate score test, and a Kullback–Leibler test. Simulation results show that these tests hold promise of detecting individual change with low Type I error and high power. A real-data example from an educational assessment illustrates the application of the proposed methods.

KW - Kullback–Leibler test

KW - individual change

KW - likelihood ratio test

KW - multidimensional item response theory

KW - score test

UR - http://www.scopus.com/inward/record.url?scp=85041501517&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85041501517&partnerID=8YFLogxK

U2 - 10.1177/0146621617726787

DO - 10.1177/0146621617726787

M3 - Article

VL - 42

SP - 221

EP - 239

JO - Applied Psychological Measurement

JF - Applied Psychological Measurement

SN - 0146-6216

IS - 3

ER -