White, T., Fraser, G. and Brown, G. orcid.org/0000-0001-8565-5476 (2019) Improving random GUI testing with image-based widget detection. In: ISSTA 2019 Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. ISSTA 2019 - 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, 15-19 Jul 2019, Beijing, China. ACM Digital Library , pp. 307-317. ISBN 9781450362245
Abstract
Graphical User Interfaces (GUIs) are amongst the most common user interfaces, enabling interactions with applications through mouse movements and key presses. Tools for automated testing of programs through their GUI exist, however they usually rely on operating system or framework specific knowledge to interact with an application. Due to frequent operating system updates, which can remove required information, and a large variety of different GUI frameworks using unique underlying data structures, such tools rapidly become obsolete, Consequently, for an automated GUI test generation tool, supporting many frameworks and operating systems is impractical. We propose a technique for improving GUI testing by automatically identifying GUI widgets in screen shots using machine learning techniques. As training data, we generate randomized GUIs to automatically extract widget information. The resulting model provides guidance to GUI testing tools in environments not currently supported by deriving GUI widget information from screen shots only. In our experiments, we found that identifying GUI widgets in screen shots and using this information to guide random testing achieved a significantly higher branch coverage in 18 of 20 applications, with an average increase of 42.5% when compared to conventional random testing.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2019 ACM. This is an author-produced version of a paper subsequently published by the ACM Digital Library. Uploaded in accordance with the publisher's self-archiving policy. |
Keywords: | GUI testing; object detection; black box testing; software engineering; data generation; convolutional neural networks |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 03 May 2019 10:00 |
Last Modified: | 10 Sep 2019 08:46 |
Status: | Published |
Publisher: | ACM Digital Library |
Refereed: | Yes |
Identification Number: | 10.1145/3293882.3330551 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:145601 |