Impact of Adaptive Feedback in Multimodal Human-Agent Interaction

Abstract

In multimodal human-agent interaction, feedback plays a critical role in guiding user behavior, with its effectiveness depending not only on the content delivered but also on how it is conveyed across modalities and agent embodiments. Feedback intervention, delivering performance-related information to guide behavior, has been widely studied in human-human interaction (HHI), where feedback levels such as task-learning, task-motivation, and self-level shape focus, motivation, and engagement; however, their role in multimodal human-agent interaction (HAI) remains underexplored. To address this, we conducted a mixed-factorial study (n = 40) evaluating real-time LLM-generated adaptive feedback across four levels (task-learning, task-motivation, self, and no feedback), two task types (promotion- and prevention-focused), and two agent embodiments (robot and voice). Our results indicate that feedback level was the primary determinant of task performance, with task-learning outperforming self-level and no-feedback, and task-motivation outperforming no-feedback. A significant three-way interaction further reveals that optimal feedback depends on both task type and agent embodiment: in promotion-focused tasks, robot-delivered task-learning feedback yields the strongest gains, whereas in prevention-focused tasks, the voice agent outperforms the robot under task-learning and no-feedback conditions. Although embodiment had no significant main effect on performance, robot-delivered feedback is perceived as significantly more useful overall. These findings highlight the importance of jointly considering feedback level, task characteristics, and agent embodiment when designing adaptive feedback in multimodal HAI.