On Seeking Moderator Variables in the Meta-Analysis of Correlational Data. A Monte Carlo Investigation of Statistical Power and Resistance to Type I Error

Paul R. Sackett, Michael M. Harris, John M. Orr

Research output: Contribution to journalArticlepeer-review

101 Scopus citations

Abstract

A series of Monte Carlo computer simulations was conducted to investigate (a) the likelihood that meta-analysis will detect true differences in effect sizes rather than attributing differences to methodological artifact and (b) the likelihood that meta-analysis will suggest the presence of moderator variables when in fact differences in effect sizes are due to methodological artifact. The simulations varied the magnitude of the true population differences between correlations, the number of studies included in the meta-analysis, and the average sample size. Simulations were run both correcting for and not correcting for measurement error. The power of three indexes-the Schmidt-Hunter ratio of expected to observed variance, the Callender-Osburn procedure, and a chi-square test-to detect true differences was investigated. Small true differences will not be detected regardless of sample size and number of studies, and moderate true differences will not be detected with small numbers of studies or small sample sizes. Hence there is a need for caution in attributing observed variation across studies to artifact.

Original languageEnglish (US)
Pages (from-to)302-310
Number of pages9
JournalJournal of Applied Psychology
Volume71
Issue number2
DOIs
StatePublished - May 1 1986

Fingerprint Dive into the research topics of 'On Seeking Moderator Variables in the Meta-Analysis of Correlational Data. A Monte Carlo Investigation of Statistical Power and Resistance to Type I Error'. Together they form a unique fingerprint.

Cite this