Automated Steering of Model-Based Test Oracles to Admit Real Program Behaviors

Gregory Gay, Sanjai Rayadurgam, Mats Heimdahl

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

The test oracle-a judge of the correctness of the system under test (SUT)-is a major component of the testing process. Specifying test oracles is challenging for some domains, such as real-time embedded systems, where small changes in timing or sensory input may cause large behavioral differences. Models of such systems, often built for analysis and simulation, are appealing for reuse as test oracles. These models, however, typically represent an idealized system, abstracting away certain issues such as non-deterministic timing behavior and sensor noise. Thus, even with the same inputs, the model's behavior may fail to match an acceptable behavior of the SUT, leading to many false positives reported by the test oracle. We propose an automated steering framework that can adjust the behavior of the model to better match the behavior of the SUT to reduce the rate of false positives. This model steering is limited by a set of constraints (defining the differences in behavior that are acceptable) and is based on a search process attempting to minimize a dissimilarity metric. This framework allows non-deterministic, but bounded, behavioral differences, while preventing future mismatches by guiding the oracle-within limits-to match the execution of the SUT. Results show that steering significantly increases SUT-oracle conformance with minimal masking of real faults and, thus, has significant potential for reducing false positives and, consequently, testing and debugging costs while improving the quality of the testing process.

Original languageEnglish (US)
Pages (from-to)531-555
Number of pages25
JournalIEEE Transactions on Software Engineering
Volume43
Issue number6
DOIs
StatePublished - Jun 1 2017

Fingerprint

Testing
Acceptance tests
Real time systems
Embedded systems
Sensors
Costs

Keywords

  • Software testing
  • model-based development
  • model-based testing
  • test oracles
  • verification

Cite this

Automated Steering of Model-Based Test Oracles to Admit Real Program Behaviors. / Gay, Gregory; Rayadurgam, Sanjai; Heimdahl, Mats.

In: IEEE Transactions on Software Engineering, Vol. 43, No. 6, 01.06.2017, p. 531-555.

Research output: Contribution to journalArticle

@article{c4d0c89ab3cc4597904df34a9e856942,
title = "Automated Steering of Model-Based Test Oracles to Admit Real Program Behaviors",
abstract = "The test oracle-a judge of the correctness of the system under test (SUT)-is a major component of the testing process. Specifying test oracles is challenging for some domains, such as real-time embedded systems, where small changes in timing or sensory input may cause large behavioral differences. Models of such systems, often built for analysis and simulation, are appealing for reuse as test oracles. These models, however, typically represent an idealized system, abstracting away certain issues such as non-deterministic timing behavior and sensor noise. Thus, even with the same inputs, the model's behavior may fail to match an acceptable behavior of the SUT, leading to many false positives reported by the test oracle. We propose an automated steering framework that can adjust the behavior of the model to better match the behavior of the SUT to reduce the rate of false positives. This model steering is limited by a set of constraints (defining the differences in behavior that are acceptable) and is based on a search process attempting to minimize a dissimilarity metric. This framework allows non-deterministic, but bounded, behavioral differences, while preventing future mismatches by guiding the oracle-within limits-to match the execution of the SUT. Results show that steering significantly increases SUT-oracle conformance with minimal masking of real faults and, thus, has significant potential for reducing false positives and, consequently, testing and debugging costs while improving the quality of the testing process.",
keywords = "Software testing, model-based development, model-based testing, test oracles, verification",
author = "Gregory Gay and Sanjai Rayadurgam and Mats Heimdahl",
year = "2017",
month = "6",
day = "1",
doi = "10.1109/TSE.2016.2615311",
language = "English (US)",
volume = "43",
pages = "531--555",
journal = "IEEE Transactions on Software Engineering",
issn = "0098-5589",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "6",

}

TY - JOUR

T1 - Automated Steering of Model-Based Test Oracles to Admit Real Program Behaviors

AU - Gay, Gregory

AU - Rayadurgam, Sanjai

AU - Heimdahl, Mats

PY - 2017/6/1

Y1 - 2017/6/1

N2 - The test oracle-a judge of the correctness of the system under test (SUT)-is a major component of the testing process. Specifying test oracles is challenging for some domains, such as real-time embedded systems, where small changes in timing or sensory input may cause large behavioral differences. Models of such systems, often built for analysis and simulation, are appealing for reuse as test oracles. These models, however, typically represent an idealized system, abstracting away certain issues such as non-deterministic timing behavior and sensor noise. Thus, even with the same inputs, the model's behavior may fail to match an acceptable behavior of the SUT, leading to many false positives reported by the test oracle. We propose an automated steering framework that can adjust the behavior of the model to better match the behavior of the SUT to reduce the rate of false positives. This model steering is limited by a set of constraints (defining the differences in behavior that are acceptable) and is based on a search process attempting to minimize a dissimilarity metric. This framework allows non-deterministic, but bounded, behavioral differences, while preventing future mismatches by guiding the oracle-within limits-to match the execution of the SUT. Results show that steering significantly increases SUT-oracle conformance with minimal masking of real faults and, thus, has significant potential for reducing false positives and, consequently, testing and debugging costs while improving the quality of the testing process.

AB - The test oracle-a judge of the correctness of the system under test (SUT)-is a major component of the testing process. Specifying test oracles is challenging for some domains, such as real-time embedded systems, where small changes in timing or sensory input may cause large behavioral differences. Models of such systems, often built for analysis and simulation, are appealing for reuse as test oracles. These models, however, typically represent an idealized system, abstracting away certain issues such as non-deterministic timing behavior and sensor noise. Thus, even with the same inputs, the model's behavior may fail to match an acceptable behavior of the SUT, leading to many false positives reported by the test oracle. We propose an automated steering framework that can adjust the behavior of the model to better match the behavior of the SUT to reduce the rate of false positives. This model steering is limited by a set of constraints (defining the differences in behavior that are acceptable) and is based on a search process attempting to minimize a dissimilarity metric. This framework allows non-deterministic, but bounded, behavioral differences, while preventing future mismatches by guiding the oracle-within limits-to match the execution of the SUT. Results show that steering significantly increases SUT-oracle conformance with minimal masking of real faults and, thus, has significant potential for reducing false positives and, consequently, testing and debugging costs while improving the quality of the testing process.

KW - Software testing

KW - model-based development

KW - model-based testing

KW - test oracles

KW - verification

UR - http://www.scopus.com/inward/record.url?scp=85027420063&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85027420063&partnerID=8YFLogxK

U2 - 10.1109/TSE.2016.2615311

DO - 10.1109/TSE.2016.2615311

M3 - Article

VL - 43

SP - 531

EP - 555

JO - IEEE Transactions on Software Engineering

JF - IEEE Transactions on Software Engineering

SN - 0098-5589

IS - 6

ER -