P value–driven methods were underpowered to detect publication bias: analysis of Cochrane review meta-analyses

Luis Furuya-Kanamori, Chang Xu, Lifeng Lin, Tinh Doan, Haitao Chu, Lukman Thalib, Suhail A.R. Doi

Research output: Contribution to journalArticle

Abstract

Objectives: The aim of the study was to investigate the effect of number of studies in a meta-analysis on the detection of publication bias using P value–driven methods. Methods: The proportion of meta-analyses detected by Egger's, Harbord's, Peters', and Begg's tests to have asymmetry suggestive of publication bias were examined in 5,014 meta-analyses from Cochrane reviews. P values were also assessed in meta-analyses with varying number of studies, whereas symmetry was held constant. A simulation study was conducted to investigate if the above tests underestimate or overestimate the presence of publication bias. Results: The proportion of meta-analyses detected as asymmetrical via Egger's, Harbord's, Peters', and Begg's tests decreased by 42.6%, 41.1%, 29.3%, and 28.3%, respectively, when the median number of studies in the meta-analysis decreased from 87 to 14. P values decreased as the number of studies increased in the meta-analysis, despite the level of symmetry remaining constant. The simulation study confirmed that when publication bias is present, P value tests underestimate the presence of publication bias, particularly when study numbers are small. Conclusion: P value–based tests used for the detection of publication bias–related asymmetry in meta-analysis require careful examination, as they underestimate asymmetry. Alternative methods not dependent on the number of studies are preferable.

Original languageEnglish (US)
Pages (from-to)86-92
Number of pages7
JournalJournal of Clinical Epidemiology
Volume118
DOIs
StatePublished - Feb 2020

Fingerprint

Publication Bias
Meta-Analysis
Publications

Keywords

  • Asymmetry
  • Meta-analysis
  • Publication bias

Cite this

P value–driven methods were underpowered to detect publication bias : analysis of Cochrane review meta-analyses. / Furuya-Kanamori, Luis; Xu, Chang; Lin, Lifeng; Doan, Tinh; Chu, Haitao; Thalib, Lukman; Doi, Suhail A.R.

In: Journal of Clinical Epidemiology, Vol. 118, 02.2020, p. 86-92.

Research output: Contribution to journalArticle

Furuya-Kanamori, Luis ; Xu, Chang ; Lin, Lifeng ; Doan, Tinh ; Chu, Haitao ; Thalib, Lukman ; Doi, Suhail A.R. / P value–driven methods were underpowered to detect publication bias : analysis of Cochrane review meta-analyses. In: Journal of Clinical Epidemiology. 2020 ; Vol. 118. pp. 86-92.
@article{62d1829343da4e8ca1da28f4b2e5189e,
title = "P value–driven methods were underpowered to detect publication bias: analysis of Cochrane review meta-analyses",
abstract = "Objectives: The aim of the study was to investigate the effect of number of studies in a meta-analysis on the detection of publication bias using P value–driven methods. Methods: The proportion of meta-analyses detected by Egger's, Harbord's, Peters', and Begg's tests to have asymmetry suggestive of publication bias were examined in 5,014 meta-analyses from Cochrane reviews. P values were also assessed in meta-analyses with varying number of studies, whereas symmetry was held constant. A simulation study was conducted to investigate if the above tests underestimate or overestimate the presence of publication bias. Results: The proportion of meta-analyses detected as asymmetrical via Egger's, Harbord's, Peters', and Begg's tests decreased by 42.6{\%}, 41.1{\%}, 29.3{\%}, and 28.3{\%}, respectively, when the median number of studies in the meta-analysis decreased from 87 to 14. P values decreased as the number of studies increased in the meta-analysis, despite the level of symmetry remaining constant. The simulation study confirmed that when publication bias is present, P value tests underestimate the presence of publication bias, particularly when study numbers are small. Conclusion: P value–based tests used for the detection of publication bias–related asymmetry in meta-analysis require careful examination, as they underestimate asymmetry. Alternative methods not dependent on the number of studies are preferable.",
keywords = "Asymmetry, Meta-analysis, Publication bias",
author = "Luis Furuya-Kanamori and Chang Xu and Lifeng Lin and Tinh Doan and Haitao Chu and Lukman Thalib and Doi, {Suhail A.R.}",
year = "2020",
month = "2",
doi = "10.1016/j.jclinepi.2019.11.011",
language = "English (US)",
volume = "118",
pages = "86--92",
journal = "Journal of Clinical Epidemiology",
issn = "0895-4356",
publisher = "Elsevier USA",

}

TY - JOUR

T1 - P value–driven methods were underpowered to detect publication bias

T2 - analysis of Cochrane review meta-analyses

AU - Furuya-Kanamori, Luis

AU - Xu, Chang

AU - Lin, Lifeng

AU - Doan, Tinh

AU - Chu, Haitao

AU - Thalib, Lukman

AU - Doi, Suhail A.R.

PY - 2020/2

Y1 - 2020/2

N2 - Objectives: The aim of the study was to investigate the effect of number of studies in a meta-analysis on the detection of publication bias using P value–driven methods. Methods: The proportion of meta-analyses detected by Egger's, Harbord's, Peters', and Begg's tests to have asymmetry suggestive of publication bias were examined in 5,014 meta-analyses from Cochrane reviews. P values were also assessed in meta-analyses with varying number of studies, whereas symmetry was held constant. A simulation study was conducted to investigate if the above tests underestimate or overestimate the presence of publication bias. Results: The proportion of meta-analyses detected as asymmetrical via Egger's, Harbord's, Peters', and Begg's tests decreased by 42.6%, 41.1%, 29.3%, and 28.3%, respectively, when the median number of studies in the meta-analysis decreased from 87 to 14. P values decreased as the number of studies increased in the meta-analysis, despite the level of symmetry remaining constant. The simulation study confirmed that when publication bias is present, P value tests underestimate the presence of publication bias, particularly when study numbers are small. Conclusion: P value–based tests used for the detection of publication bias–related asymmetry in meta-analysis require careful examination, as they underestimate asymmetry. Alternative methods not dependent on the number of studies are preferable.

AB - Objectives: The aim of the study was to investigate the effect of number of studies in a meta-analysis on the detection of publication bias using P value–driven methods. Methods: The proportion of meta-analyses detected by Egger's, Harbord's, Peters', and Begg's tests to have asymmetry suggestive of publication bias were examined in 5,014 meta-analyses from Cochrane reviews. P values were also assessed in meta-analyses with varying number of studies, whereas symmetry was held constant. A simulation study was conducted to investigate if the above tests underestimate or overestimate the presence of publication bias. Results: The proportion of meta-analyses detected as asymmetrical via Egger's, Harbord's, Peters', and Begg's tests decreased by 42.6%, 41.1%, 29.3%, and 28.3%, respectively, when the median number of studies in the meta-analysis decreased from 87 to 14. P values decreased as the number of studies increased in the meta-analysis, despite the level of symmetry remaining constant. The simulation study confirmed that when publication bias is present, P value tests underestimate the presence of publication bias, particularly when study numbers are small. Conclusion: P value–based tests used for the detection of publication bias–related asymmetry in meta-analysis require careful examination, as they underestimate asymmetry. Alternative methods not dependent on the number of studies are preferable.

KW - Asymmetry

KW - Meta-analysis

KW - Publication bias

UR - http://www.scopus.com/inward/record.url?scp=85076027542&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85076027542&partnerID=8YFLogxK

U2 - 10.1016/j.jclinepi.2019.11.011

DO - 10.1016/j.jclinepi.2019.11.011

M3 - Article

C2 - 31743750

AN - SCOPUS:85076027542

VL - 118

SP - 86

EP - 92

JO - Journal of Clinical Epidemiology

JF - Journal of Clinical Epidemiology

SN - 0895-4356

ER -