The Relative Power of Training Evaluation Designs Under Different Cost Configurations

After noting that the statistical power of training evaluation designs is a complex function of sample size, the reliability of the dependent measure, the correlation between pre- and posttest measures, and whether a randomized pretest-posttest or randomized posttest-only design is used, we show that the costs of conducting an evaluation are important considerations that also affect the relative power of the designs. Specifically, subject costs, administrative costs, and item development costs are different components that can absorb resources when training evaluations are conducted. When total cost resources are fixed, these separate costs affect the relative power of pretest-posttest and posttest-only designs differently, and the posttest-only design may be the more preferred design under many different conditions. In other words, a variety of design and parameter tradeoffs affect power when total costs are fixed.