Methods matter: A multi-trait multi-method analysis of student behavior

Faith G. Miller, Austin H. Johnson, Huihui Yu, Sandra M. Chafouleas, D. Betsy McCoach, T. Chris Riley-Tillman, Gregory A. Fabiano, Megan E. Welsh

Research output: Contribution to journalArticle

3 Citations (Scopus)

Abstract

Reliable and valid data form the foundation for evidence-based practices, yet surprisingly few studies on school-based behavioral assessments have been conducted which implemented one of the most fundamental approaches to construct validation, the multitrait-multimethod matrix (MTMM). To this end, the current study examined the reliability and validity of data derived from three commonly utilized school-based behavioral assessment methods: Direct Behavior Rating – Single Item Scales, systematic direct observations, and behavior rating scales on three common constructs of interest: academically engaged, disruptive, and respectful behavior. Further, this study included data from different sources including student self-report, teacher report, and external observers. A total of 831 students in grades 3–8 and 129 teachers served as participants. Data were analyzed using bivariate correlations of the MTMM, as well as single and multi-level structural equation modeling. Results suggested the presence of strong methods effects for all the assessment methods utilized, as well as significant relations between constructs of interest. Implications for practice and future research are discussed.

Original languageEnglish (US)
Pages (from-to)53-72
Number of pages20
JournalJournal of School Psychology
Volume68
DOIs
StatePublished - Jun 2018

Fingerprint

Students
student
Evidence-Based Practice
Information Storage and Retrieval
Reproducibility of Results
Self Report
teacher
rating scale
school
school grade
rating
evidence
Problem Behavior
Behavior Rating Scale

Keywords

  • Assessment
  • Behavior
  • Direct Behavior Rating
  • MTMM
  • Rating scales
  • Systematic direct observation
  • Validity

Cite this

Miller, F. G., Johnson, A. H., Yu, H., Chafouleas, S. M., McCoach, D. B., Riley-Tillman, T. C., ... Welsh, M. E. (2018). Methods matter: A multi-trait multi-method analysis of student behavior. Journal of School Psychology, 68, 53-72. https://doi.org/10.1016/j.jsp.2018.01.002

Methods matter : A multi-trait multi-method analysis of student behavior. / Miller, Faith G.; Johnson, Austin H.; Yu, Huihui; Chafouleas, Sandra M.; McCoach, D. Betsy; Riley-Tillman, T. Chris; Fabiano, Gregory A.; Welsh, Megan E.

In: Journal of School Psychology, Vol. 68, 06.2018, p. 53-72.

Research output: Contribution to journalArticle

Miller, FG, Johnson, AH, Yu, H, Chafouleas, SM, McCoach, DB, Riley-Tillman, TC, Fabiano, GA & Welsh, ME 2018, 'Methods matter: A multi-trait multi-method analysis of student behavior', Journal of School Psychology, vol. 68, pp. 53-72. https://doi.org/10.1016/j.jsp.2018.01.002
Miller, Faith G. ; Johnson, Austin H. ; Yu, Huihui ; Chafouleas, Sandra M. ; McCoach, D. Betsy ; Riley-Tillman, T. Chris ; Fabiano, Gregory A. ; Welsh, Megan E. / Methods matter : A multi-trait multi-method analysis of student behavior. In: Journal of School Psychology. 2018 ; Vol. 68. pp. 53-72.
@article{79e96157cc24482384b321b3bcd0f34d,
title = "Methods matter: A multi-trait multi-method analysis of student behavior",
abstract = "Reliable and valid data form the foundation for evidence-based practices, yet surprisingly few studies on school-based behavioral assessments have been conducted which implemented one of the most fundamental approaches to construct validation, the multitrait-multimethod matrix (MTMM). To this end, the current study examined the reliability and validity of data derived from three commonly utilized school-based behavioral assessment methods: Direct Behavior Rating – Single Item Scales, systematic direct observations, and behavior rating scales on three common constructs of interest: academically engaged, disruptive, and respectful behavior. Further, this study included data from different sources including student self-report, teacher report, and external observers. A total of 831 students in grades 3–8 and 129 teachers served as participants. Data were analyzed using bivariate correlations of the MTMM, as well as single and multi-level structural equation modeling. Results suggested the presence of strong methods effects for all the assessment methods utilized, as well as significant relations between constructs of interest. Implications for practice and future research are discussed.",
keywords = "Assessment, Behavior, Direct Behavior Rating, MTMM, Rating scales, Systematic direct observation, Validity",
author = "Miller, {Faith G.} and Johnson, {Austin H.} and Huihui Yu and Chafouleas, {Sandra M.} and McCoach, {D. Betsy} and Riley-Tillman, {T. Chris} and Fabiano, {Gregory A.} and Welsh, {Megan E.}",
year = "2018",
month = "6",
doi = "10.1016/j.jsp.2018.01.002",
language = "English (US)",
volume = "68",
pages = "53--72",
journal = "Journal of School Psychology",
issn = "0022-4405",
publisher = "Elsevier BV",

}

TY - JOUR

T1 - Methods matter

T2 - A multi-trait multi-method analysis of student behavior

AU - Miller, Faith G.

AU - Johnson, Austin H.

AU - Yu, Huihui

AU - Chafouleas, Sandra M.

AU - McCoach, D. Betsy

AU - Riley-Tillman, T. Chris

AU - Fabiano, Gregory A.

AU - Welsh, Megan E.

PY - 2018/6

Y1 - 2018/6

N2 - Reliable and valid data form the foundation for evidence-based practices, yet surprisingly few studies on school-based behavioral assessments have been conducted which implemented one of the most fundamental approaches to construct validation, the multitrait-multimethod matrix (MTMM). To this end, the current study examined the reliability and validity of data derived from three commonly utilized school-based behavioral assessment methods: Direct Behavior Rating – Single Item Scales, systematic direct observations, and behavior rating scales on three common constructs of interest: academically engaged, disruptive, and respectful behavior. Further, this study included data from different sources including student self-report, teacher report, and external observers. A total of 831 students in grades 3–8 and 129 teachers served as participants. Data were analyzed using bivariate correlations of the MTMM, as well as single and multi-level structural equation modeling. Results suggested the presence of strong methods effects for all the assessment methods utilized, as well as significant relations between constructs of interest. Implications for practice and future research are discussed.

AB - Reliable and valid data form the foundation for evidence-based practices, yet surprisingly few studies on school-based behavioral assessments have been conducted which implemented one of the most fundamental approaches to construct validation, the multitrait-multimethod matrix (MTMM). To this end, the current study examined the reliability and validity of data derived from three commonly utilized school-based behavioral assessment methods: Direct Behavior Rating – Single Item Scales, systematic direct observations, and behavior rating scales on three common constructs of interest: academically engaged, disruptive, and respectful behavior. Further, this study included data from different sources including student self-report, teacher report, and external observers. A total of 831 students in grades 3–8 and 129 teachers served as participants. Data were analyzed using bivariate correlations of the MTMM, as well as single and multi-level structural equation modeling. Results suggested the presence of strong methods effects for all the assessment methods utilized, as well as significant relations between constructs of interest. Implications for practice and future research are discussed.

KW - Assessment

KW - Behavior

KW - Direct Behavior Rating

KW - MTMM

KW - Rating scales

KW - Systematic direct observation

KW - Validity

UR - http://www.scopus.com/inward/record.url?scp=85041801173&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85041801173&partnerID=8YFLogxK

U2 - 10.1016/j.jsp.2018.01.002

DO - 10.1016/j.jsp.2018.01.002

M3 - Article

C2 - 29861031

AN - SCOPUS:85041801173

VL - 68

SP - 53

EP - 72

JO - Journal of School Psychology

JF - Journal of School Psychology

SN - 0022-4405

ER -