Parry, O., Kapfhammer, G.M., Hilton, M. et al. (1 more author) (2022) Surveying the developer experience of flaky tests. In: ICSE-SEIP '22: Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice. ICSE '22: 44th International Conference on Software Engineering, 21-29 May 2022, Pennsylvania, Pittsburgh. Association for Computing Machinery (ACM) , pp. 253-262. ISBN: 9781450392266 ISSN: 0270-5257 EISSN: 1558-1225
Abstract
Test cases that pass and fail without changes to the code under test are known as flaky. The past decade has seen increasing research interest in flaky tests, though little attention has been afforded to the views and experiences of software developers. In this study, we utilized a multi-source approach to obtain insights into how developers define flaky tests, their experiences of the impacts and causes of flaky tests, and the actions they take in response to them. To that end, we designed a literature-guided developer survey that we deployed on social media, receiving 170 total responses. We also searched on StackOverflow and analyzed 38 threads relevant to flaky tests, offering a distinct perspective free of any self-reporting bias. Through a mixture of numerical and thematic analyses, this study reveals a number of findings, including (1) developers strongly agree that flaky tests hinder continuous integration; (2) developers who experience flaky tests more often may be more likely to ignore potentially genuine test failures; and (3) developers rate issues in setup and teardown to be the most common causes of flaky tests.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2022 ACM. This is an author-produced version of a paper subsequently published in ICSE-SEIP '22: Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice. Uploaded in accordance with the publisher's self-archiving policy. |
Keywords: | Software Testing; Flaky Tests; Qualitative Research |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
Funding Information: | Funder Grant number META PLATFORM INC 2537673289630253 ENGINEERING AND PHYSICAL SCIENCE RESEARCH COUNCIL EP/T015764/1 |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 06 Aug 2025 14:17 |
Last Modified: | 06 Aug 2025 14:18 |
Status: | Published |
Publisher: | Association for Computing Machinery (ACM) |
Refereed: | Yes |
Identification Number: | 10.1145/3510457.3513037 |
Related URLs: | |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:230090 |