Test order in teacher-rated behavior assessments: Is counterbalancing necessary

Janice Kooken, Megan E. Welsh, D. Betsy McCoach, Faith G. Miller, Sandra M. Chafouleas, T. Chris Riley-Tillman, Gregory Fabiano

Research output: Contribution to journalArticle

3 Citations (Scopus)

Abstract

Counterbalancing treatment order in experimental research design is well established as an option to reduce threats to internal validity, but in educational and psychological research, the effect of varying the order of multiple tests to a single rater has not been examined and is rarely adhered to in practice. The current study examines the effect of test order on measures of student behavior by teachers as raters utilizing data from a behavior measure validation study. Using multilevel modeling to control for students nested within teachers, the effect of rating an earlier measure on the intercept or slope of a later behavior assessment was statistically significant in 22% of predictor main effects for the spring test period. Test order effects had potential for high stakes consequences with differences large enough to change risk classification. Results suggest that researchers and practitioners in classroom settings using multiple measures evaluate the potential impact of test order. Where possible, they should counterbalance when the risk of an order effect exists and report justification for the decision to not counterbalance.

Original languageEnglish (US)
Pages (from-to)98-109
Number of pages12
JournalPsychological assessment
Volume29
Issue number1
DOIs
StatePublished - Jan 1 2017

Fingerprint

Research Design
Students
Validation Studies
Research Personnel
Psychology
Research
Therapeutics

Keywords

  • Counterbalancing
  • Teacher raters
  • Test order effects

Cite this

Kooken, J., Welsh, M. E., McCoach, D. B., Miller, F. G., Chafouleas, S. M., Riley-Tillman, T. C., & Fabiano, G. (2017). Test order in teacher-rated behavior assessments: Is counterbalancing necessary. Psychological assessment, 29(1), 98-109. https://doi.org/10.1037/pas0000314

Test order in teacher-rated behavior assessments : Is counterbalancing necessary. / Kooken, Janice; Welsh, Megan E.; McCoach, D. Betsy; Miller, Faith G.; Chafouleas, Sandra M.; Riley-Tillman, T. Chris; Fabiano, Gregory.

In: Psychological assessment, Vol. 29, No. 1, 01.01.2017, p. 98-109.

Research output: Contribution to journalArticle

Kooken, J, Welsh, ME, McCoach, DB, Miller, FG, Chafouleas, SM, Riley-Tillman, TC & Fabiano, G 2017, 'Test order in teacher-rated behavior assessments: Is counterbalancing necessary', Psychological assessment, vol. 29, no. 1, pp. 98-109. https://doi.org/10.1037/pas0000314
Kooken J, Welsh ME, McCoach DB, Miller FG, Chafouleas SM, Riley-Tillman TC et al. Test order in teacher-rated behavior assessments: Is counterbalancing necessary. Psychological assessment. 2017 Jan 1;29(1):98-109. https://doi.org/10.1037/pas0000314
Kooken, Janice ; Welsh, Megan E. ; McCoach, D. Betsy ; Miller, Faith G. ; Chafouleas, Sandra M. ; Riley-Tillman, T. Chris ; Fabiano, Gregory. / Test order in teacher-rated behavior assessments : Is counterbalancing necessary. In: Psychological assessment. 2017 ; Vol. 29, No. 1. pp. 98-109.
@article{99b03092c1424e50875dad3d323a434d,
title = "Test order in teacher-rated behavior assessments: Is counterbalancing necessary",
abstract = "Counterbalancing treatment order in experimental research design is well established as an option to reduce threats to internal validity, but in educational and psychological research, the effect of varying the order of multiple tests to a single rater has not been examined and is rarely adhered to in practice. The current study examines the effect of test order on measures of student behavior by teachers as raters utilizing data from a behavior measure validation study. Using multilevel modeling to control for students nested within teachers, the effect of rating an earlier measure on the intercept or slope of a later behavior assessment was statistically significant in 22{\%} of predictor main effects for the spring test period. Test order effects had potential for high stakes consequences with differences large enough to change risk classification. Results suggest that researchers and practitioners in classroom settings using multiple measures evaluate the potential impact of test order. Where possible, they should counterbalance when the risk of an order effect exists and report justification for the decision to not counterbalance.",
keywords = "Counterbalancing, Teacher raters, Test order effects",
author = "Janice Kooken and Welsh, {Megan E.} and McCoach, {D. Betsy} and Miller, {Faith G.} and Chafouleas, {Sandra M.} and Riley-Tillman, {T. Chris} and Gregory Fabiano",
year = "2017",
month = "1",
day = "1",
doi = "10.1037/pas0000314",
language = "English (US)",
volume = "29",
pages = "98--109",
journal = "Psychological Assessment",
issn = "1040-3590",
publisher = "American Psychological Association",
number = "1",

}

TY - JOUR

T1 - Test order in teacher-rated behavior assessments

T2 - Is counterbalancing necessary

AU - Kooken, Janice

AU - Welsh, Megan E.

AU - McCoach, D. Betsy

AU - Miller, Faith G.

AU - Chafouleas, Sandra M.

AU - Riley-Tillman, T. Chris

AU - Fabiano, Gregory

PY - 2017/1/1

Y1 - 2017/1/1

N2 - Counterbalancing treatment order in experimental research design is well established as an option to reduce threats to internal validity, but in educational and psychological research, the effect of varying the order of multiple tests to a single rater has not been examined and is rarely adhered to in practice. The current study examines the effect of test order on measures of student behavior by teachers as raters utilizing data from a behavior measure validation study. Using multilevel modeling to control for students nested within teachers, the effect of rating an earlier measure on the intercept or slope of a later behavior assessment was statistically significant in 22% of predictor main effects for the spring test period. Test order effects had potential for high stakes consequences with differences large enough to change risk classification. Results suggest that researchers and practitioners in classroom settings using multiple measures evaluate the potential impact of test order. Where possible, they should counterbalance when the risk of an order effect exists and report justification for the decision to not counterbalance.

AB - Counterbalancing treatment order in experimental research design is well established as an option to reduce threats to internal validity, but in educational and psychological research, the effect of varying the order of multiple tests to a single rater has not been examined and is rarely adhered to in practice. The current study examines the effect of test order on measures of student behavior by teachers as raters utilizing data from a behavior measure validation study. Using multilevel modeling to control for students nested within teachers, the effect of rating an earlier measure on the intercept or slope of a later behavior assessment was statistically significant in 22% of predictor main effects for the spring test period. Test order effects had potential for high stakes consequences with differences large enough to change risk classification. Results suggest that researchers and practitioners in classroom settings using multiple measures evaluate the potential impact of test order. Where possible, they should counterbalance when the risk of an order effect exists and report justification for the decision to not counterbalance.

KW - Counterbalancing

KW - Teacher raters

KW - Test order effects

UR - http://www.scopus.com/inward/record.url?scp=84964786209&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84964786209&partnerID=8YFLogxK

U2 - 10.1037/pas0000314

DO - 10.1037/pas0000314

M3 - Article

C2 - 27099978

AN - SCOPUS:84964786209

VL - 29

SP - 98

EP - 109

JO - Psychological Assessment

JF - Psychological Assessment

SN - 1040-3590

IS - 1

ER -