Foster, M. orcid.org/0000-0001-8233-9873, Wild, C., Hierons, R. et al. (1 more author) (Accepted: 2023) Causal test adequacy. In: 2024 IEEE Conference on Software Testing, Verification and Validation (ICST). 17th IEEE International Conference on Software Testing, Verification and Validation (ICST) 2024, 27-31 May 2024, Toronto, Canada. Institute of Electrical and Electronics Engineers (IEEE) . (In Press)
Abstract
Causal reasoning is becoming an increasingly popular technique for testing software. In this setting, the tester starts from a simple directed graph that captures their underlying understanding of causal relationships between relevant variables in the program, and this knowledge is then used to reason about causal input-output relationships that are observed during testing. One question that has not yet been addressed in this context is how to measure test adequacy: How do we know whether a causal relationship (or set of relationships) has been properly established by a test set? In this paper we present a metric inspired by Weyuker’s notion of inference adequacy. For a given causal relationship, we estimate the causal effect from the test data. The basis of our adequacy metric is then an estimate of the convergence of this estimate, which we calculate using statistical bootstrapping. We evaluate our metric on tests for three diverse computational models. The results show a statistically significant correlation between our metric and a test suite’s ability to detect mutants, and also that it is a good indicator of whether a sufficient number of system executions have been observed to trust the outcome of the test.
Metadata
Authors/Creators: |
|
||||
---|---|---|---|---|---|
Copyright, Publisher and Additional Information: | © 2024 The Author(s). | ||||
Keywords: | software testing; causal inference; test adequacy | ||||
Dates: |
|
||||
Institution: | The University of Sheffield | ||||
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) | ||||
Funding Information: |
|
||||
Depositing User: | Symplectic Sheffield | ||||
Date Deposited: | 02 Feb 2024 16:01 | ||||
Last Modified: | 02 Feb 2024 16:01 | ||||
Status: | In Press | ||||
Publisher: | Institute of Electrical and Electronics Engineers (IEEE) | ||||
Refereed: | Yes | ||||
Related URLs: |
Download
Filename: main.pdf