Automated Oracle Data Selection Support

Gregory Gay, Matt Staats, Michael Whalen, Mats P.E. Heimdahl

Research output: Contribution to journalArticlepeer-review

17 Scopus citations


The choice of test oracle - the artifact that determines whether an application under test executes correctly - can significantly impact the effectiveness of the testing process. However, despite the prevalence of tools that support test input selection, little work exists for supporting oracle creation. We propose a method of supporting test oracle creation that automatically selects the oracle data - the set of variables monitored during testing - for expected value test oracles. This approach is based on the use of mutation analysis to rank variables in terms of fault-finding effectiveness, thus automating the selection of the oracle data. Experimental results obtained by employing our method over six industrial systems (while varying test input types and the number of generated mutants) indicate that our method - when paired with test inputs generated either at random or to satisfy specific structural coverage criteria - may be a cost-effective approach for producing small, effective oracle data sets, with fault finding improvements over current industrial best practice of up to 1,435 percent observed (with typical improvements of up to 50 percent).

Original languageEnglish (US)
Article number7112189
Pages (from-to)1119-1137
Number of pages19
JournalIEEE Transactions on Software Engineering
Issue number11
StatePublished - Nov 1 2015

Bibliographical note

Publisher Copyright:
© 2015 IEEE.


  • Testing
  • oracle data
  • oracle selection
  • test oracles
  • verification


Dive into the research topics of 'Automated Oracle Data Selection Support'. Together they form a unique fingerprint.

Cite this