Robust sequential experimental strategy for black-box optimization with application to hyperparameter tuning

Global optimization of expensive, noisy black-box functions is a problem of significant interest and is used extensively in applications such as hyperparameter optimization (HPO) of machine learning models. Response Surface Optimization (RSO) and Bayesian Optimization (BO) are two highly efficient model-based experimentation strategies for optimizing expensive, noisy black-box functions. The run-size efficiency of both RSO and BO is critically dependent on the fitness of the surrogate model in approximating the true response function. In this regard, both RSO and BO deliberately choose initial designs that are optimal for estimating the assumed surrogate model and their sequential design strategy is aimed at further refining the surrogate model estimate. As a result, a design strategy that is optimal for one is not necessarily optimal for the other. In this study, we propose Compound-RSO, a model-robust, three-stage experimental strategy for optimizing noisy black-box functions with continuous experimental factors. Compound-RSO is a hybrid strategy between RSO and BO which empirically estimates the complexity of the black-box function and chooses the appropriate strategy between them. We propose robust designs that are highly efficient for estimating a second-order polynomial model and that simultaneously maximize the power of the Lack-of-Fit test to determine the inadequacies of a second-order approximation. Additionally, we propose a trust-region-based-adaptive experimentation strategy for optimizing a second-order response function. Through our simulation study we illustrate that the proposed Compound-RSO strategy is more efficient than BO when the response function is second-order, and performs comparably to BO when the response function is complex. Lastly, a case study on HPO of Deep Neural Networks (DDNs) in the context of visual quality inspections at a medical device manufacturer further illustrates the utility of the proposed Compound-RSO strategy.