Predictive vision-language monitoring for proactive safety in robot task execution

Abstract

Robots that execute language-conditioned tasks in dynamic environments often rely on feedback only after an action has failed, which can be insufficient when failures involve collisions or workspace conflicts. This paper presents a predictive monitoring framework that uses Vision-Language Models (VLMs) to assess near-future execution risk during robot task execution. The framework first generates structured plans with action execution conditions and a plan-level fallback action. During execution, a monitoring module combines visual observations, the current action, and the relevant execution conditions to estimate whether a condition is likely to be violated within a short future time window. When the predicted risk exceeds a task-specific threshold, the robot halts the current action, executes the fallback behavior, and replans from the updated state. We evaluate the approach in Gazebo simulation on mobile navigation with a moving human obstacle and manipulation with two robot arms sharing a workspace. Across controlled collision-risk settings, the proposed method achieves higher task success rates than reactive VLM-based baselines while requiring fewer replanning events than conservative current-state precondition checking. The results indicate that predictive vision-language monitoring can improve task completion in simulated dynamic robot tasks, while remaining subject to limitations such as VLM latency, prompt sensitivity, and evaluation beyond simulation.