Evaluation of the type I error rate when using parametric bootstrap analysis of a cluster randomized controlled trial with binary outcomes and a small number of clusters

Lilian Golzarri-Arroyo; Stephanie L. Dickinson; Yasaman Jamshidi-Naeini; Roger S. Zoh; Andrew W. Brown; Arthur H. Owora; Peng Li; J. Michael Oakes; David B. Allison

doi:10.1016/j.cmpb.2022.106654

Evaluation of the type I error rate when using parametric bootstrap analysis of a cluster randomized controlled trial with binary outcomes and a small number of clusters

Lilian Golzarri-Arroyo
, Stephanie L. Dickinson
, Yasaman Jamshidi-Naeini
, Roger S. Zoh
, Andrew W. Brown
, Arthur H. Owora
, Peng Li
, J. Michael Oakes
, David B. Allison

Administration (RIO)

Research output: Contribution to journal › Article › peer-review

7 Scopus citations

Abstract

Background: Cluster randomized controlled trials (cRCTs) are increasingly used but must be analyzed carefully. We conducted a simulation study to evaluate the validity of a parametric bootstrap (PB) approach with respect to the empirical type I error rate for a cRCT with binary outcomes and a small number of clusters. Methods: We simulated a case study with a binary (0/1) outcome, four clusters, and 100 subjects per cluster. To compare the validity of the test with respect to error rate, we simulated the same experiment with K=10, 20, and 30 clusters, each with 2,000 simulated datasets. To test the null hypothesis, we used a generalized linear mixed model including a random intercept for clusters and obtained p-values based on likelihood ratio tests (LRTs) using the parametric bootstrap method as implemented in the R package “pbkrtest”. Results: The PB test produced error rates of 9.1%, 5.5%, 4.9%, and 5.0% on average across all ICC values for K=4, K=10, K=20, and K=30, respectively. The error rates were higher, ranging from 9.1% to 36.5% for K=4, in the models with singular fits (i.e., ignoring clustering) because the ICC was estimated to be zero.

Original language	English (US)
Article number	106654
Journal	Computer Methods and Programs in Biomedicine
Volume	215
DOIs	https://doi.org/10.1016/j.cmpb.2022.106654
State	Published - Mar 2022

Bibliographical note

Funding Information:
The computation of this research was performed on IU's Big Red 3 supercomputer, which was supported in part by Lilly Endowment, Inc., through its support for the Indiana University Pervasive Technology Institute. We thank Jennifer Holmes for language editing. We are grateful to Dr. Keith Muller and Dr. Pengcheng Xun for their constructive comments.

Funding Information:
This work was supported by the National Institutes of Health (NIH) [grant numbers R25HL124208, R25DK099080, R25GM141507, P30AG050886, and U24AG056053]. The opinions expressed are those of the authors and do not necessarily represent those of the NIH or any other organization.

Publisher Copyright:
© 2022 Elsevier B.V.

Keywords

Cluster Analysis
Computer Simulation
Humans
Linear Models
Research Design
Sample Size

PubMed: MeSH publication types

Journal Article
Randomized Controlled Trial

Publisher link

10.1016/j.cmpb.2022.106654

Find It

Find It at U of M Libraries

Cite this

Golzarri-Arroyo, L., Dickinson, S. L., Jamshidi-Naeini, Y., Zoh, R. S., Brown, A. W., Owora, A. H., Li, P., Oakes, J. M., & Allison, D. B. (2022). Evaluation of the type I error rate when using parametric bootstrap analysis of a cluster randomized controlled trial with binary outcomes and a small number of clusters. Computer Methods and Programs in Biomedicine, 215, Article 106654. https://doi.org/10.1016/j.cmpb.2022.106654

Evaluation of the type I error rate when using parametric bootstrap analysis of a cluster randomized controlled trial with binary outcomes and a small number of clusters. / Golzarri-Arroyo, Lilian; Dickinson, Stephanie L.; Jamshidi-Naeini, Yasaman et al.
In: Computer Methods and Programs in Biomedicine, Vol. 215, 106654, 03.2022.

Research output: Contribution to journal › Article › peer-review

Golzarri-Arroyo, L, Dickinson, SL, Jamshidi-Naeini, Y, Zoh, RS, Brown, AW, Owora, AH, Li, P, Oakes, JM & Allison, DB 2022, 'Evaluation of the type I error rate when using parametric bootstrap analysis of a cluster randomized controlled trial with binary outcomes and a small number of clusters', Computer Methods and Programs in Biomedicine, vol. 215, 106654. https://doi.org/10.1016/j.cmpb.2022.106654

@article{83cd939e4f4e46e984202de9cb073fd0,

title = "Evaluation of the type I error rate when using parametric bootstrap analysis of a cluster randomized controlled trial with binary outcomes and a small number of clusters",

abstract = "Background: Cluster randomized controlled trials (cRCTs) are increasingly used but must be analyzed carefully. We conducted a simulation study to evaluate the validity of a parametric bootstrap (PB) approach with respect to the empirical type I error rate for a cRCT with binary outcomes and a small number of clusters. Methods: We simulated a case study with a binary (0/1) outcome, four clusters, and 100 subjects per cluster. To compare the validity of the test with respect to error rate, we simulated the same experiment with K=10, 20, and 30 clusters, each with 2,000 simulated datasets. To test the null hypothesis, we used a generalized linear mixed model including a random intercept for clusters and obtained p-values based on likelihood ratio tests (LRTs) using the parametric bootstrap method as implemented in the R package “pbkrtest”. Results: The PB test produced error rates of 9.1\%, 5.5\%, 4.9\%, and 5.0\% on average across all ICC values for K=4, K=10, K=20, and K=30, respectively. The error rates were higher, ranging from 9.1\% to 36.5\% for K=4, in the models with singular fits (i.e., ignoring clustering) because the ICC was estimated to be zero.",

keywords = "Cluster Analysis, Computer Simulation, Humans, Linear Models, Research Design, Sample Size",

author = "Lilian Golzarri-Arroyo and Dickinson, \{Stephanie L.\} and Yasaman Jamshidi-Naeini and Zoh, \{Roger S.\} and Brown, \{Andrew W.\} and Owora, \{Arthur H.\} and Peng Li and Oakes, \{J. Michael\} and Allison, \{David B.\}",

note = "Funding Information: The computation of this research was performed on IU's Big Red 3 supercomputer, which was supported in part by Lilly Endowment, Inc., through its support for the Indiana University Pervasive Technology Institute. We thank Jennifer Holmes for language editing. We are grateful to Dr. Keith Muller and Dr. Pengcheng Xun for their constructive comments. Funding Information: This work was supported by the National Institutes of Health (NIH) [grant numbers R25HL124208, R25DK099080, R25GM141507, P30AG050886, and U24AG056053]. The opinions expressed are those of the authors and do not necessarily represent those of the NIH or any other organization. Publisher Copyright: {\textcopyright} 2022 Elsevier B.V.",

year = "2022",

month = mar,

doi = "10.1016/j.cmpb.2022.106654",

language = "English (US)",

volume = "215",

journal = "Computer Methods and Programs in Biomedicine",

issn = "0169-2607",

publisher = "Elsevier Ireland Ltd",

}

TY - JOUR

T1 - Evaluation of the type I error rate when using parametric bootstrap analysis of a cluster randomized controlled trial with binary outcomes and a small number of clusters

AU - Golzarri-Arroyo, Lilian

AU - Dickinson, Stephanie L.

AU - Jamshidi-Naeini, Yasaman

AU - Zoh, Roger S.

AU - Brown, Andrew W.

AU - Owora, Arthur H.

AU - Li, Peng

AU - Oakes, J. Michael

AU - Allison, David B.

N1 - Funding Information: The computation of this research was performed on IU's Big Red 3 supercomputer, which was supported in part by Lilly Endowment, Inc., through its support for the Indiana University Pervasive Technology Institute. We thank Jennifer Holmes for language editing. We are grateful to Dr. Keith Muller and Dr. Pengcheng Xun for their constructive comments. Funding Information: This work was supported by the National Institutes of Health (NIH) [grant numbers R25HL124208, R25DK099080, R25GM141507, P30AG050886, and U24AG056053]. The opinions expressed are those of the authors and do not necessarily represent those of the NIH or any other organization. Publisher Copyright: © 2022 Elsevier B.V.

PY - 2022/3

Y1 - 2022/3

N2 - Background: Cluster randomized controlled trials (cRCTs) are increasingly used but must be analyzed carefully. We conducted a simulation study to evaluate the validity of a parametric bootstrap (PB) approach with respect to the empirical type I error rate for a cRCT with binary outcomes and a small number of clusters. Methods: We simulated a case study with a binary (0/1) outcome, four clusters, and 100 subjects per cluster. To compare the validity of the test with respect to error rate, we simulated the same experiment with K=10, 20, and 30 clusters, each with 2,000 simulated datasets. To test the null hypothesis, we used a generalized linear mixed model including a random intercept for clusters and obtained p-values based on likelihood ratio tests (LRTs) using the parametric bootstrap method as implemented in the R package “pbkrtest”. Results: The PB test produced error rates of 9.1%, 5.5%, 4.9%, and 5.0% on average across all ICC values for K=4, K=10, K=20, and K=30, respectively. The error rates were higher, ranging from 9.1% to 36.5% for K=4, in the models with singular fits (i.e., ignoring clustering) because the ICC was estimated to be zero.

AB - Background: Cluster randomized controlled trials (cRCTs) are increasingly used but must be analyzed carefully. We conducted a simulation study to evaluate the validity of a parametric bootstrap (PB) approach with respect to the empirical type I error rate for a cRCT with binary outcomes and a small number of clusters. Methods: We simulated a case study with a binary (0/1) outcome, four clusters, and 100 subjects per cluster. To compare the validity of the test with respect to error rate, we simulated the same experiment with K=10, 20, and 30 clusters, each with 2,000 simulated datasets. To test the null hypothesis, we used a generalized linear mixed model including a random intercept for clusters and obtained p-values based on likelihood ratio tests (LRTs) using the parametric bootstrap method as implemented in the R package “pbkrtest”. Results: The PB test produced error rates of 9.1%, 5.5%, 4.9%, and 5.0% on average across all ICC values for K=4, K=10, K=20, and K=30, respectively. The error rates were higher, ranging from 9.1% to 36.5% for K=4, in the models with singular fits (i.e., ignoring clustering) because the ICC was estimated to be zero.

KW - Cluster Analysis

KW - Computer Simulation

KW - Humans

KW - Linear Models

KW - Research Design

KW - Sample Size

UR - https://www.scopus.com/pages/publications/85123605260

UR - https://www.scopus.com/pages/publications/85123605260#tab=citedBy

U2 - 10.1016/j.cmpb.2022.106654

DO - 10.1016/j.cmpb.2022.106654

M3 - Article

C2 - 35093646

AN - SCOPUS:85123605260

SN - 0169-2607

VL - 215

JO - Computer Methods and Programs in Biomedicine

JF - Computer Methods and Programs in Biomedicine

M1 - 106654

ER -

Evaluation of the type I error rate when using parametric bootstrap analysis of a cluster randomized controlled trial with binary outcomes and a small number of clusters

Abstract

Bibliographical note

Keywords

PubMed: MeSH publication types

Publisher link

Find It

Other files and links

Fingerprint

Cite this