Several test adequacy criteria have been developed for quantifying the the coverage of deep neural networks (DNNs) achieved by a test suite. Being dependent on the structure of the DNN, these can be costly to measure and use, especially given the highly iterative nature of the model training workflow. Further, testing provides higher overall assurance when such implementation dependent measures are used along with implementation independent ones. In this paper, we rigorously define a new black-box coverage criterion that is independent of the DNN model under test. We further describe a few desirable properties and associated evaluation metrics for assessing test coverage criteria and use those to empirically compare and contrast the black-box criterion with several DNN structural coverage criteria. Results indicate that the black-box criterion has comparable effectiveness and provides benefits that complement white-box criteria. The results also reveal a few weaknesses of coverage criteria for DNNs.
|Original language||English (US)|
|Title of host publication||Proceedings - 2021 IEEE 32nd International Symposium on Software Reliability Engineering, ISSRE 2021|
|Editors||Zhi Jin, Xuandong Li, Jianwen Xiang, Leonardo Mariani, Ting Liu, Xiao Yu, Nahgmeh Ivaki|
|Publisher||IEEE Computer Society|
|Number of pages||12|
|State||Published - 2021|
|Event||32nd IEEE International Symposium on Software Reliability Engineering, ISSRE 2021 - Wuhan, China|
Duration: Oct 25 2021 → Oct 28 2021
|Name||Proceedings - International Symposium on Software Reliability Engineering, ISSRE|
|Conference||32nd IEEE International Symposium on Software Reliability Engineering, ISSRE 2021|
|Period||10/25/21 → 10/28/21|
Bibliographical noteFunding Information:
This work was supported by AFRL and DARPA under contract FA8750-18-C-0099.
© 2021 IEEE.