Cross-Behavior Learning with Object Flow Prediction for Robotic Manipulation

Cross-embodiment learning enables robots to acquire manipulation skills by learning from demonstrations provided by different embodiments. However, most existing research on cross-embodiment learning focuses on transferring similar manipulation behaviors. For the same task, different embodiments may need different behaviors; e.g., it may be easier for a human to push an object to a goal position, while a robot may use a pick-and-place to perform the same task. Making use of such cross-embodiment and cross-behavior demonstrations becomes essential for large-scale imitation learning. In this work, we propose a novel framework for cross-behavior learning based on object flow. Object flow represents the task in a manner that is independent from the embodiment and the particular behavior used during the demonstration. By shifting the focus from the manipulator to objects, our framework enables learning from cross-behavior manipulation data rather than merely imitating the behavior of a specific embodiment. Our results on task representations show that, with only 20 robot demonstrations, integrating object flow prediction improves success rates by up to 91% in-domain and 87% out-of-domain. Adding 40 human demonstrations in addition to the 20 robot demonstrations further boosts out-of-domain performance by 143%.