Fraser, G. and Walkinshaw, N. (2015) Assessing and generating test sets in terms of behavioural adequacy. Software Testing Verification and Reliability, 25 (8). pp. 749-780. ISSN 0960-0833
Abstract
Identifying a finite test set that adequately captures the essential behaviour of a program such that all faults are identified is a well-established problem. This is traditionally addressed with syntactic adequacy metrics (e.g. branch coverage), but these can be impractical and may be misleading even if they are satisfied. One intuitive notion of adequacy, which has been discussed in theoretical terms over the past three decades, is the idea of behavioural coverage: If it is possible to infer an accurate model of a system from its test executions, then the test set can be deemed to be adequate. Despite its intuitive basis, it has remained almost entirely in the theoretical domain because inferred models have been expected to be exact (generally an infeasible task) and have not allowed for any pragmatic interim measures of adequacy to guide test set generation. This paper presents a practical approach to incorporate behavioural coverage. Our BESTEST approach (1) enables the use of machine learning algorithms to augment standard syntactic testing approaches and (2) shows how search-based testing techniques can be applied to generate test sets with respect to this criterion. An empirical study on a selection of Java units demonstrates that test sets with higher behavioural coverage significantly outperform current baseline test criteria in terms of detected faults.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2015 The Authors. Software Testing, Verification and Reliability published by John Wiley & Sons, Ltd. This is an open access article under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits use, distribution and reproduction in any medium, provided the original work is properly cited. |
Keywords: | test generation; test adequacy; search-based software testing |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
Funding Information: | Funder Grant number GOOGLE UNSPECIFIED ENGINEERING AND PHYSICAL SCIENCE RESEARCH COUNCIL (EPSRC) EP/K030353/1 |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 22 Feb 2016 14:41 |
Last Modified: | 22 Feb 2016 14:41 |
Published Version: | http://dx.doi.org/10.1002/stvr.1575 |
Status: | Published |
Publisher: | Wiley |
Refereed: | Yes |
Identification Number: | 10.1002/stvr.1575 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:95332 |