Statistical methods for computer performance evaluation

Computer performance has traditionally played a fundamental role in computer system design. Performance measurements and their quantitative analyses are useful in understanding and enhancing a computer system and in comparing different systems. Owing to the stochastic nature of systems, measuring performance is prone to noise and experimental errors. The field of statistics provides a rich set of tools and techniques to effectively deal with such noise and errors, allowing one to obtain meaningful measurements. Statistical techniques have also been applied in many other areas of computer performance measurement, such as designing experiments, simulation, analyzing results, and predicting performance. In this article, we present a comprehensive discussion of how statistical theories apply during the process of computer performance measurement and describe the statistical techniques that have been used in the literature.