A permutation procedure to detect heterogeneous treatment effects in randomized clinical trials while controlling the type I error rate

Research output: Contribution to journalArticlepeer-review

1 Scopus citations


Background/Aims: Secondary analyses of randomized clinical trials often seek to identify subgroups with differential treatment effects. These discoveries can help guide individual treatment decisions based on patient characteristics and identify populations for which additional treatments are needed. Traditional analyses require researchers to pre-specify potential subgroups to reduce the risk of reporting spurious results. There is a need for methods that can detect such subgroups without a priori specification while allowing researchers to control the probability of falsely detecting heterogeneous subgroups when treatment effects are uniform across the study population. Methods: We propose a permutation procedure for tuning parameter selection that allows for type I error control when testing for heterogeneous treatment effects framed within the Virtual Twins procedure for subgroup identification. We verify that the type I error rate can be controlled at the nominal rate and investigate the power for detecting heterogeneous effects when present through extensive simulation studies. We apply our method to a secondary analysis of data from a randomized trial of very low nicotine content cigarettes. Results: In the absence of type I error control, the observed type I error rate for Virtual Twins was between 99% and 100%. In contrast, models tuned via the proposed permutation were able to control the type I error rate and detect heterogeneous effects when present. An application of our approach to a recently completed trial of very low nicotine content cigarettes identified several variables with potentially heterogeneous treatment effects. Conclusions: The proposed permutation procedure allows researchers to engage in secondary analyses of clinical trials for treatment effect heterogeneity while maintaining the type I error rate without pre-specifying subgroups.

Original languageEnglish (US)
Pages (from-to)512-521
Number of pages10
JournalClinical Trials
Issue number5
StatePublished - Oct 2022

Bibliographical note

Funding Information:
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by the National Cancer Institute (Award Numbers R01CA214825 and R01CA22 5190), the National Institute on Drug Abuse (Award Numbers R01DA046320, and U54-DA031659) and National Center for Advancing Translational Science (Award Number UL1TR002494). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or Food and Drug Administration.

Publisher Copyright:
© The Author(s) 2022.


  • Permutation test
  • subgroup identification
  • treatment effect heterogeneity
  • type I error
  • virtual twins


Dive into the research topics of 'A permutation procedure to detect heterogeneous treatment effects in randomized clinical trials while controlling the type I error rate'. Together they form a unique fingerprint.

Cite this