Vision-based Gesture Tracking for Teleoperating Mobile Manipulators

In unconstrained environments, teleoperation exists in consumer-level robots. Motion capture technology has been shown to shrink the skill gap between end user and teleoperation of robots, however, the equipment is getting more expensive and complex such as with motion capture suits. This paper provides a vision-based method with a camera to reduce costs and make teleoperation accessible for consumers. The hand coordinates received from the camera are estimated through Google MediaPipe, a hand posture estimation package. Teleoperation strategies are composed of coordinates and hand gestures. We conduct a simulation study on the Husky robotic platform with a manipulator in PyBullet and demonstrate the control framework through various gestures.