Counterfactual fairness for small subgroups

While methods for measuring and correcting differential performance in risk prediction models haveproliferated in recent years, most existing techniques can only be used to assess fairness across relativelylarge subgroups. The purpose of algorithmic fairness efforts is often to redress discrimination against groups that are both marginalized and small, so this sample size limitation can prevent existing techniques from accomplishing their main aim. In clinical applications, this challenge combines with statistical issues that arise when models are used to guide treatment. We take a 3-step approach to addressing both of these challenges, building on the “counterfactual fairness” framework that accounts for confounding by treatment. First, we propose new estimands that leverage information across groups. Second, we estimate these quantities using a larger volume of data than existing techniques. Finally, we propose a novel data borrowing approach to incorporate “external data” that lacks outcomes and predictions but contains covariate and group membership information. We demonstrate application of our estimators to a risk prediction model used by a major Midwestern health system during the coronavirus disease 2019 (COVID-19) pandemic.