Fraser, G., McMinn, P., Staats, M. et al. (2 more authors) (2013) Does automated white-box test generation really help software testers? 2013 International Symposium on Software Testing and Analysis, ISSTA 2013 - Proceedings. 291 - 301.
Abstract
Automated test generation techniques can efficiently produce test data that systematically cover structural aspects of a program. In the absence of a specification, a common assumption is that these tests relieve a developer of most of the work, as the act of testing is reduced to checking the results of the tests. Although this assumption has persisted for decades, there has been no conclusive evidence to date confirming it. However, the fact that the approach has only seen a limited uptake in industry suggests the contrary, and calls into question its practical usefulness. To investigate this issue, we performed a controlled experiment comparing a total of 49 subjects split between writing tests manually and writing tests with the aid of an automated unit test generation tool, EvoSuite. We found that, on one hand, tool support leads to clear improvements in commonly applied quality metrics such as code coverage (up to 300% increase). However, on the other hand, there was no measurable improvement in the number of bugs actually found by developers. Our results not only cast some doubt on how the research community evaluates test generation tools, but also point to improvements and future work necessary before automated test generation tools will be widely adopted by practitioners. © 2013 ACM.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2013 The Author(s). This is an Open Access article distributed under the terms of the Creative Commons Attribution Licence (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
Funding Information: | Funder Grant number EPSRC EP/I010386/1 |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 03 Jul 2014 15:00 |
Last Modified: | 03 Jul 2014 15:00 |
Published Version: | http://dx.doi.org/10.1145/2483760.2483774 |
Status: | Published |
Publisher: | ACM |
Refereed: | Yes |
Identification Number: | 10.1145/2483760.2483774 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:79647 |