INPERAL: Interactive Perception Alignment in Human-Robot Interaction

Abstract

Embodied AI aims to develop agents capable of perceiving, acting, and collaborating with humans in dynamic environments. However, existing Vision-Language Model (VLM)-based approaches to Human-Robot Interaction (HRI) often assume accurate shared understanding and depend on task-specific finetuning, limiting robustness and generalization. In practice, both human instructions and robot perceptions are inherently error-prone, leading to misalignment that can significantly degrade task performance. To address this challenge, we propose INPERAL, an interactive framework that detects and resolves perception mismatches between humans and embodied agents using pretrained VLMs. INPERAL incorporates a Perception Error Detector to identify inconsistencies between human intent and agent perception, and leverages interactive conversation alongside environment exploration to iteratively refine understanding and resolve these errors. Once alignment is achieved, INPERAL generates an executable action plan to complete the task. Evaluations in two real-world HRI scenarios with diverse perception misalignment settings demonstrate that INPERAL effectively mitigates misalignment, improves task success rates, and enhances robustness without requiring task-specific finetuning, underscoring the importance of interactive error resolution for reliable HRI.