Testing one hypothesis multiple times

In applied settings, hypothesis testing when a nuisance parameter is identifiable only under the alternative often reduces to a problem of testing one hypothesis multiple times (TOHM). Specifically, a fine discretization of the space of the nonidentifiable parameter is specified, and the null hypothesis is tested against a set of sub-alternative hypotheses, one for each point of the discretization. The resulting sub-test statistics are then combined to obtain a global p-value. We propose a computationally efficient inferential tool to perform TOHM under stringent significance requirements, such as those typically required in the physical sciences, (e.g., a p-value < 10^-7). The resulting procedure leads to a generalized approach to performing inferences under nonstandard conditions, including non-nested model comparisons.