Moving the goalposts: Coverage satisfaction is not enough

Structural coverage criteria have been proposed to measure the adequacy of testing efforts. Indeed, in some domains-e.g., critical systems areas-structural coverage criteria must be satisfied to achieve certification. The advent of powerful search-based test generation tools has given us the ability to generate test inputs to satisfy these structural coverage criteria. While tempting, recent empirical evidence indicates these tools should be used with caution, as merely achieving high structural coverage is not necessarily indicative of high fault detection ability. In this report, we review some of these findings, and offer recommendations on how the strengths of search-based test generation methods can alleviate these issues.