Paterson, D., Kapfhammer, G.M., Gordon, F. et al. (1 more author) (2018) Using Controlled Numbers of Real Faults and Mutants to Empirically Evaluate Coverage-Based Test Case Prioritization. In: Proceedings of the International Workshop on Automation of Software Test (AST 2018). International Workshop on Automation of Software Test (AST 2018), 28-29 May 2018, Gothenburg, Sweden. ACM , New york , pp. 57-63. ISBN 978-1-4503-5743-2
Abstract
Used to establish confidence in the correctness of evolving software, regression testing is an important, yet costly, task. Test case prioritization enables the rapid detection of faults during regression testing by reordering the test suite so that effective tests are run as early as is possible. However, a distinct lack of information about the regression faults found in complex real-world software forced prior experimental studies of these methods to use artificial faults called mutants. Using the Defects4J database of real faults, this paper presents the results of experiments evaluating the effectiveness of four representative test prioritization techniques. Since this paper's results show that prioritization is susceptible to high amounts of variance when only one fault is present, our experiments also control the number of real faults and mutants in the program subject to regression testing. Our overall findings are that, in comparison to mutants, real faults are harder for reordered test suites to quickly detect, suggesting that mutants are not a surrogate for real faults.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2018 ACM. This is an author produced version of a paper subsequently published in Proceedings of the 13th International Workshop on Automation of Software Test. Uploaded in accordance with the publisher's self-archiving policy. |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 30 May 2018 15:10 |
Last Modified: | 19 Dec 2022 13:49 |
Published Version: | https://doi.org/10.1145/3194733.3194735 |
Status: | Published online |
Publisher: | ACM |
Refereed: | Yes |
Identification Number: | 10.1145/3194733.3194735 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:131042 |