Foundation Model-Driven Imitation Learning for Robots Without Manual Object Specification

Abstract

A central challenge in collaborative robotics is enabling robots to acquire new skills directly from human demonstrations without requiring prior knowledge of the objects involved. We present a system that records user activities and decomposes them into primitive actions by analyzing hand trajectories, gaze-based attentional cues, and object interactions in three-dimensional space. We present the segmentation algorithm in detail, highlighting how objects are automatically segmented and semantically identified through the use of foundation models, while hand pose estimation ensures accurate capture of grasp configurations. Moreover, primitives such as grasping, moving, interacting, and releasing are extracted, each corresponding to distinct, temporally segmented motion patterns in 3D Cartesian space. These primitives are subsequently composed to represent complex activities, enabling the robot to reproduce them with objects placed in new configurations or locations. This approach minimizes user intervention and leverages advances in vision and language foundation models, moving toward more adaptive and scalable imitation learning in collaborative environments.