A comparison of permutation and mixed-model regression methods for the analysis of simulated data in the context of a group-randomized trial

Our first purpose was to determine whether, in the context of a group-randomized trial (GRT) with Gaussian errors, permutation or mixed-model regression methods fare better in the presence of measurable confounding in terms of their Monte Carlo type I error rates and power. Our results indicate that given a proper randomization, the type I error rate is similar for both methods, whether unadjusted or adjusted, even in small studies. However, our results also show that should the investigator face the unfortunate circumstance in which modest confounding exists in the only realization available, the unadjusted analysis risks a type I error; in this regard, there was little to distinguish the two methods. Finally, our results show that power is similar for the two methods and, not surprisingly, better for the adjusted tests. Our second purpose was to examine the relative performa nce of permutation and mixed-model regression methods in the context of a GRT when the normality assumptions underlying the mixed model are violated. Published studies have examined the impact of violation of this assumption at the member level only. Our findings indicate that both methods perform well when the assumption is violated so long as the ICC is very small and the design is balanced at the group level. However, at ICC ≥0.01, the permutation test carries the nominal type I error rate while the model-based test is conservative and so less powerful. Binomial group- and member-level errors did not otherwise change the relative performance of the two methods with regard to confounding.