Modelling hand gestures to test leap motion controlled applications

Programs that use a Natural User Interface (NUI) are not controlled with a mouse and keyboard, but through input devices that monitor the user’s body movements. Manually testing applications through such interfaces is time-consuming. Generating realistic test data automatically is also challenging, because the input is a complex data structure that represents real body structures and movements. Previously, it has been shown that models learned from user interactions can be used to generate tests for NUI applications controlled by the Microsoft Kinect. In this paper, we study the case of the Leap Motion input device, which allows applications to be controlled with hand movements and finger positions, resulting in substantially more complex input data structures. We present a framework to model human hand data interacting with applications, and generate test data automatically from these models. We also evaluate the influence of the training data, as well as the influence of using a single model of the complete user data vs. multiple models for the different aspects of hand movement (e.g., finger positions, hand positions, hand rotations). Experiments on five applications controlled by the Leap Motion demonstrate that our approach generates effective test data. The quality and quantity of the training data used to derive the models is the main factor that determines their effectiveness. On the other hand, the effects of using multiple (as opposed to single) models are minor and application specific.