Real-world Performance of Meta-analysis Methods for Double-Zero-Event Studies with Dichotomous Outcomes Using the Cochrane Database of Systematic Reviews

Yanan Ren, Lifeng Lin, Qinshu Lian, Hui Zou, Haitao Chu

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

Background: Meta-analysis combines multiple independent studies, which can increase power and provide better estimates. However, it is unclear how best to deal with studies with zero events; such studies are also known as double-zero-event studies (DZS). Several statistical methods have been proposed, but the agreement among different approaches has not been systematically assessed using real-world published systematic reviews. Methods: The agreement of five commonly used methods (i.e., the inverse-variance, Mantel–Haenszel, Peto, Bayesian, and exact methods) was assessed using the Cohen’s κ coefficients using 368 meta-analyses with rare events selected from the Cochrane Database of Systematic Reviews. Three continuity corrections, including the correction of a constant 0.5, the treatment arm continuity correction (TACC), and the empirical (EMP) correction, were used to handle DZS when applying inverse-variance and Mantel–Haenszel methods. Results: When the proportion of DZS studies was lower than 50% in a meta-analysis, different methods had moderately high agreement. However, when this proportion was increased to be over 50%, the agreement among the methods decreased to different extents. For the Bayesian, exact, and Peto methods and the inverse-variance and Mantel–Haenszel methods using the EMP correction, their agreement coefficients with the inverse-variance and Mantel–Haenszel methods using a constant 0.5 and TACC decreased from larger than 0.70 to smaller than 0.30. In contrast, the agreement coefficients only decreased slightly among the Bayesian, exact, and Peto methods and the inverse-variance and Mantel–Haenszel methods using the EMP correction. Conclusions: To utilize all available information and reduce research waste and avoid overestimating the effect, meta-analysts should incorporate DZS, rather than simply removing them. The Peto and other conventional methods with continuity correction should be avoided when the proportion of DZS is extremely high. The exact and Bayesian methods are highly recommended, except when none of the included studies have an event in one or both treatment arms.

Original languageEnglish (US)
JournalJournal of general internal medicine
DOIs
StatePublished - Jan 1 2019

Fingerprint

Meta-Analysis
Databases
Bayes Theorem

Keywords

  • Cochrane Database of Systematic Reviews
  • Cohen’s κ coefficient
  • double-zero-event studies
  • meta-analysis
  • rare event

PubMed: MeSH publication types

  • Journal Article

Cite this

@article{c7e8574616354ecfb8197fbad218ff3a,
title = "Real-world Performance of Meta-analysis Methods for Double-Zero-Event Studies with Dichotomous Outcomes Using the Cochrane Database of Systematic Reviews",
abstract = "Background: Meta-analysis combines multiple independent studies, which can increase power and provide better estimates. However, it is unclear how best to deal with studies with zero events; such studies are also known as double-zero-event studies (DZS). Several statistical methods have been proposed, but the agreement among different approaches has not been systematically assessed using real-world published systematic reviews. Methods: The agreement of five commonly used methods (i.e., the inverse-variance, Mantel–Haenszel, Peto, Bayesian, and exact methods) was assessed using the Cohen’s κ coefficients using 368 meta-analyses with rare events selected from the Cochrane Database of Systematic Reviews. Three continuity corrections, including the correction of a constant 0.5, the treatment arm continuity correction (TACC), and the empirical (EMP) correction, were used to handle DZS when applying inverse-variance and Mantel–Haenszel methods. Results: When the proportion of DZS studies was lower than 50{\%} in a meta-analysis, different methods had moderately high agreement. However, when this proportion was increased to be over 50{\%}, the agreement among the methods decreased to different extents. For the Bayesian, exact, and Peto methods and the inverse-variance and Mantel–Haenszel methods using the EMP correction, their agreement coefficients with the inverse-variance and Mantel–Haenszel methods using a constant 0.5 and TACC decreased from larger than 0.70 to smaller than 0.30. In contrast, the agreement coefficients only decreased slightly among the Bayesian, exact, and Peto methods and the inverse-variance and Mantel–Haenszel methods using the EMP correction. Conclusions: To utilize all available information and reduce research waste and avoid overestimating the effect, meta-analysts should incorporate DZS, rather than simply removing them. The Peto and other conventional methods with continuity correction should be avoided when the proportion of DZS is extremely high. The exact and Bayesian methods are highly recommended, except when none of the included studies have an event in one or both treatment arms.",
keywords = "Cochrane Database of Systematic Reviews, Cohen’s κ coefficient, double-zero-event studies, meta-analysis, rare event",
author = "Yanan Ren and Lifeng Lin and Qinshu Lian and Hui Zou and Haitao Chu",
year = "2019",
month = "1",
day = "1",
doi = "10.1007/s11606-019-04925-8",
language = "English (US)",
journal = "Journal of General Internal Medicine",
issn = "0884-8734",
publisher = "Springer New York LLC",

}

TY - JOUR

T1 - Real-world Performance of Meta-analysis Methods for Double-Zero-Event Studies with Dichotomous Outcomes Using the Cochrane Database of Systematic Reviews

AU - Ren, Yanan

AU - Lin, Lifeng

AU - Lian, Qinshu

AU - Zou, Hui

AU - Chu, Haitao

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Background: Meta-analysis combines multiple independent studies, which can increase power and provide better estimates. However, it is unclear how best to deal with studies with zero events; such studies are also known as double-zero-event studies (DZS). Several statistical methods have been proposed, but the agreement among different approaches has not been systematically assessed using real-world published systematic reviews. Methods: The agreement of five commonly used methods (i.e., the inverse-variance, Mantel–Haenszel, Peto, Bayesian, and exact methods) was assessed using the Cohen’s κ coefficients using 368 meta-analyses with rare events selected from the Cochrane Database of Systematic Reviews. Three continuity corrections, including the correction of a constant 0.5, the treatment arm continuity correction (TACC), and the empirical (EMP) correction, were used to handle DZS when applying inverse-variance and Mantel–Haenszel methods. Results: When the proportion of DZS studies was lower than 50% in a meta-analysis, different methods had moderately high agreement. However, when this proportion was increased to be over 50%, the agreement among the methods decreased to different extents. For the Bayesian, exact, and Peto methods and the inverse-variance and Mantel–Haenszel methods using the EMP correction, their agreement coefficients with the inverse-variance and Mantel–Haenszel methods using a constant 0.5 and TACC decreased from larger than 0.70 to smaller than 0.30. In contrast, the agreement coefficients only decreased slightly among the Bayesian, exact, and Peto methods and the inverse-variance and Mantel–Haenszel methods using the EMP correction. Conclusions: To utilize all available information and reduce research waste and avoid overestimating the effect, meta-analysts should incorporate DZS, rather than simply removing them. The Peto and other conventional methods with continuity correction should be avoided when the proportion of DZS is extremely high. The exact and Bayesian methods are highly recommended, except when none of the included studies have an event in one or both treatment arms.

AB - Background: Meta-analysis combines multiple independent studies, which can increase power and provide better estimates. However, it is unclear how best to deal with studies with zero events; such studies are also known as double-zero-event studies (DZS). Several statistical methods have been proposed, but the agreement among different approaches has not been systematically assessed using real-world published systematic reviews. Methods: The agreement of five commonly used methods (i.e., the inverse-variance, Mantel–Haenszel, Peto, Bayesian, and exact methods) was assessed using the Cohen’s κ coefficients using 368 meta-analyses with rare events selected from the Cochrane Database of Systematic Reviews. Three continuity corrections, including the correction of a constant 0.5, the treatment arm continuity correction (TACC), and the empirical (EMP) correction, were used to handle DZS when applying inverse-variance and Mantel–Haenszel methods. Results: When the proportion of DZS studies was lower than 50% in a meta-analysis, different methods had moderately high agreement. However, when this proportion was increased to be over 50%, the agreement among the methods decreased to different extents. For the Bayesian, exact, and Peto methods and the inverse-variance and Mantel–Haenszel methods using the EMP correction, their agreement coefficients with the inverse-variance and Mantel–Haenszel methods using a constant 0.5 and TACC decreased from larger than 0.70 to smaller than 0.30. In contrast, the agreement coefficients only decreased slightly among the Bayesian, exact, and Peto methods and the inverse-variance and Mantel–Haenszel methods using the EMP correction. Conclusions: To utilize all available information and reduce research waste and avoid overestimating the effect, meta-analysts should incorporate DZS, rather than simply removing them. The Peto and other conventional methods with continuity correction should be avoided when the proportion of DZS is extremely high. The exact and Bayesian methods are highly recommended, except when none of the included studies have an event in one or both treatment arms.

KW - Cochrane Database of Systematic Reviews

KW - Cohen’s κ coefficient

KW - double-zero-event studies

KW - meta-analysis

KW - rare event

UR - http://www.scopus.com/inward/record.url?scp=85063212338&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85063212338&partnerID=8YFLogxK

U2 - 10.1007/s11606-019-04925-8

DO - 10.1007/s11606-019-04925-8

M3 - Article

C2 - 30887438

AN - SCOPUS:85063212338

JO - Journal of General Internal Medicine

JF - Journal of General Internal Medicine

SN - 0884-8734

ER -