On the feasibility and challenges of synthesizing executable Espresso tests

Several tools have been proposed to automatically test Android applications, achieving outstanding results in terms of both code coverage and crash discovery. While useful for crash reproduction and bug-fixing, these tools usually do not present the generated interactions in a format that motivates developers to read and modify such tests later on. This hinders the ability of developers to add those tests to their existing test suites, or adapt them to new scenarios- common practices in modern software development where tests are maintained and evolve alongside production code. In this work we present an empirical study on the challenges of automatically synthesizing Espresso test suites from sequences of interactions over widgets. We build on top of the MATE testing tool and implement a prototype that enables this study. The prototype is then evaluated on 12 open-source Android apps, followed by an analysis and discussion of challenges and limitations. We also include feedback from developers of open-source projects and an industrial app. Our empirical study shows that the creation of Espresso tests is difficult, mostly due to the lack of unique properties to unambiguously identify specific widgets in the UI. This problem is aggravated in some cases by the incomplete or ambiguous definition of GUI components and layouts. It also points out that further research is needed to find ways to improve the testability of Android apps either manually or automatically. Nonetheless, the feedback received indicates that the synthesized Espresso tests are still useful for projects with few or no test cases, serving as a starting point for creating new ones.