P value–driven methods were underpowered to detect publication bias: analysis of Cochrane review meta-analyses

Luis Furuya-Kanamori, Chang Xu, Lifeng Lin, Tinh Doan, Haitao Chu, Lukman Thalib, Suhail A.R. Doi

Research output: Contribution to journalArticlepeer-review

14 Scopus citations


Objectives: The aim of the study was to investigate the effect of number of studies in a meta-analysis on the detection of publication bias using P value–driven methods. Methods: The proportion of meta-analyses detected by Egger's, Harbord's, Peters', and Begg's tests to have asymmetry suggestive of publication bias were examined in 5,014 meta-analyses from Cochrane reviews. P values were also assessed in meta-analyses with varying number of studies, whereas symmetry was held constant. A simulation study was conducted to investigate if the above tests underestimate or overestimate the presence of publication bias. Results: The proportion of meta-analyses detected as asymmetrical via Egger's, Harbord's, Peters', and Begg's tests decreased by 42.6%, 41.1%, 29.3%, and 28.3%, respectively, when the median number of studies in the meta-analysis decreased from 87 to 14. P values decreased as the number of studies increased in the meta-analysis, despite the level of symmetry remaining constant. The simulation study confirmed that when publication bias is present, P value tests underestimate the presence of publication bias, particularly when study numbers are small. Conclusion: P value–based tests used for the detection of publication bias–related asymmetry in meta-analysis require careful examination, as they underestimate asymmetry. Alternative methods not dependent on the number of studies are preferable.

Original languageEnglish (US)
Pages (from-to)86-92
Number of pages7
JournalJournal of Clinical Epidemiology
StatePublished - Feb 2020

Bibliographical note

Funding Information:
Funding: This work was supported by Program Grant #NPRP10-0129-170274 from the Qatar National Research Fund (a member of Qatar Foundation). The findings herein reflect the work and are solely the responsibility of the authors. All authors had full access to all the data in the study, and the corresponding author had final responsibility for the decision to submit for publication. L.F.-K. was supported by an Australian National Health and Medical Research Council Fellowship (APP1158469).


  • Asymmetry
  • Meta-analysis
  • Publication bias

PubMed: MeSH publication types

  • Journal Article
  • Research Support, Non-U.S. Gov't

Fingerprint Dive into the research topics of 'P value–driven methods were underpowered to detect publication bias: analysis of Cochrane review meta-analyses'. Together they form a unique fingerprint.

Cite this