The effect of program and model structure on the effectiveness of MC/DC test adequacy coverage

Gregory Gay, Ajitha Rajan, Matt Staats, Michael Whalen, Mats P.E. Heimdahl

Research output: Contribution to journalArticlepeer-review

19 Scopus citations

Abstract

Test adequacy metrics defined over the structure of a program, such as Modified Condition and Decision Coverage (MC/DC), are used to assess testing efforts. However, MC/DC can be "cheated" by restructuring a program to make it easier to achieve the desired coverage. This is concerning, given the importance of MC/DC in assessing the adequacy of test suites for critical systems domains. In this work, we have explored the impact of implementation structure on the efficacy of test suites satisfying the MC/DC criterion using four real-world avionics systems. Our results demonstrate that test suites achieving MC/DC over implementations with structurally complex Boolean expressions are generally larger and more effective than test suites achieving MC/DC over functionally equivalent, but structurally simpler, implementations. Additionally, we found that test suites generated over simpler implementations achieve significantly lower MC/DC and fault-finding effectiveness when applied to complex implementations, whereas test suites generated over the complex implementation still achieve high MC/DC and attain high fault finding over the simpler implementation. By measuring MC/DC over simple implementations, we can significantly reduce the cost of testing, but in doing so, we also reduce the effectiveness of the testing process. Thus, developers have an economic incentive to "cheat" the MC/DC criterion, but this cheating leads to negative consequences. Accordingly, we recommend that organizations require MC/DC over a structurally complex implementation for testing purposes to avoid these consequences.

Original languageEnglish (US)
Article number25
JournalACM Transactions on Software Engineering and Methodology
Volume25
Issue number3
DOIs
StatePublished - Jul 2016

Bibliographical note

Funding Information:
Gregory Gay, Ajitha Rajan, Matt Staats, Michael Whalen, and Mats P. E. Heimdahl. 2016. The effect of program and model structure on the effectiveness of MC/DC test adequacy coverage. ACM Trans. Softw. Eng. Methodol. 25, 3, Article 25 (July 2016), 34 pages. DOI: http://dx.doi.org/10.1145/2934672 This work has been partially supported by NASA Ames Research Center Cooperative Agreement NNA06CB21A, NASA IV&V Facility Contract NNG-05CB16C, NSF grants CNS-0931931 and CNS-1035715, and the L-3 Titan Group. Authors’ addresses: G. Gay, Department of Computer Science & Engineering, University of South Carolina, Columbia, SC, USA, 29208; email: greg@greggay.com; A. Rajan, School of Informatics, University of Edinburgh, Edinburgh, UK, EH1 2QL; email: ajitha.rajan@gmail.com; M. Staats, Google. Inc., Zurich, Switzerland; email: staatsm@gmail.com; M. Whalen and M. P. E. Heimdahl, Department of Computer Science & Engineering, University of Minnesota, Minneapolis, MN, USA, 55455; emails: {whalen, heimdahl}@cs. umn.edu. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies show this notice on the first page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701 USA, fax +1 (212) 869-0481, or permissions@acm.org. ©c 2016 ACM 1049-331X/2016/07-ART25 $15.00 DOI: http://dx.doi.org/10.1145/2934672

Publisher Copyright:
© 2016 ACM.

Keywords

  • Coverage
  • Fault finding

Fingerprint Dive into the research topics of 'The effect of program and model structure on the effectiveness of MC/DC test adequacy coverage'. Together they form a unique fingerprint.

Cite this