A stepwise approach to software-hardware performance co-optimization using design of experiments

Configurations of hardware and software parameters play a major role in determining the performance of an application program. With increasingly complex systems, the task of finding the optimal configuration parameters becomes very difficult When a performance analyst needs to find the best hardware and software settings to run an application, the traditional approach of trying different combinations, such as trial-and-error and tuning one factor at-a-time, might not lead to optimal results. We present a methodology for applying Design of Experiments (DOE) techniques to vary the software and hardware factors in a systematic manner. We show how to use a Plackett and Burman (PB) analysis to find the main factors and then a full-factorial design followed by Analysis of variance (ANOVA) and the F-test to statistically quantify the effects of the main parameters and interactions, This systematic approach reduces the time and cost needed to select the software and hardware configurations to optimize a software application on a new or existing platform. We demonstrate with two case studies how this approach can be applied to software-hardware co-optimization. After using the proposed approach, nearly 8 times improvement in response time was seen for one of the in-house benchmarks, and 23.6% improvement in throughput was seen for SPECjvm2008, as compared to the baseline one-factor at-a-time approach. In all experiments, this systematic approach outperforms the traditional approach. We hope to share our excitement about the opportunities DOE provides to reduce the time taken for performing hardware and software tuning to improve application performance and human productivity.