Robust sequential experimental strategy for black-box optimization with application to hyperparameter tuning

Gautham Sunder, Thomas A. Albrecht, Christopher J. Nachtsheim

Research output: Contribution to journalArticlepeer-review


Global optimization of expensive, noisy black-box functions is a problem of significant interest and is used extensively in applications such as hyperparameter optimization (HPO) of machine learning models. Response Surface Optimization (RSO) and Bayesian Optimization (BO) are two highly efficient model-based experimentation strategies for optimizing expensive, noisy black-box functions. The run-size efficiency of both RSO and BO is critically dependent on the fitness of the surrogate model in approximating the true response function. In this regard, both RSO and BO deliberately choose initial designs that are optimal for estimating the assumed surrogate model and their sequential design strategy is aimed at further refining the surrogate model estimate. As a result, a design strategy that is optimal for one is not necessarily optimal for the other. In this study, we propose Compound-RSO, a model-robust, three-stage experimental strategy for optimizing noisy black-box functions with continuous experimental factors. Compound-RSO is a hybrid strategy between RSO and BO which empirically estimates the complexity of the black-box function and chooses the appropriate strategy between them. We propose robust designs that are highly efficient for estimating a second-order polynomial model and that simultaneously maximize the power of the Lack-of-Fit test to determine the inadequacies of a second-order approximation. Additionally, we propose a trust-region-based-adaptive experimentation strategy for optimizing a second-order response function. Through our simulation study we illustrate that the proposed Compound-RSO strategy is more efficient than BO when the response function is second-order, and performs comparably to BO when the response function is complex. Lastly, a case study on HPO of Deep Neural Networks (DDNs) in the context of visual quality inspections at a medical device manufacturer further illustrates the utility of the proposed Compound-RSO strategy.

Original languageEnglish (US)
Pages (from-to)3992-4014
Number of pages23
JournalQuality and Reliability Engineering International
Issue number8
StatePublished - Dec 2022

Bibliographical note

Funding Information:
The authors thank Boston Scientific Corporation for their support; the propriety data and AI library shared by the data science team at Boston Scientific Corporation was critical to this work.

Publisher Copyright:
© 2022 The Authors. Quality and Reliability Engineering International published by John Wiley & Sons Ltd.


  • Bayesian optimization
  • compound optimal designs
  • hyperparameter optimization
  • noisy computer experiments
  • response surface optimization


Dive into the research topics of 'Robust sequential experimental strategy for black-box optimization with application to hyperparameter tuning'. Together they form a unique fingerprint.

Cite this