Stuck-Assist: A Large Multimodal Model-Based Stuck Recovery Assist for Driving on Off-Road Deformable Terrains

Abstract

Recent advancements in off-road autonomy have shown significant progress in perception, planning, and control frameworks, including end-to-end learning approaches. Comprehensive results have been demonstrated in both simulation and real-world experiments; however, there are significant challenges in critical cases that need further evaluation. One such challenge is the immobilization of autonomous ground vehicles (AGVs) in unstructured off-road environments, which can significantly impact agriculture, space exploration, military operations, and search and rescue missions. Addressing this problem requires recovery strategies that are context-sensitive, adaptable to terrain and vehicle conditions, and effective in integrating multimodal inputs. To this end, this paper investigates the use of a large multimodal model (LMM) providing higher-level planning assistance with human-in-the-loop evaluations for vehicle recovery after immobilization in unstructured off-road terrain. The experimental simulation platform developed was based on the Algoryx (AGX) Dynamics engine for high-fidelity terramechanics interaction and vehicle physics combined with Unreal Engine 5. This platform was further integrated with a driving simulator equipped with steering wheel and pedal interfaces for human-in-the-loop experiments. We evaluated ten representative unstuck scenarios across two deformable terrains (loose sand and compact sand) under two modes: an unskilled baseline, where participants attempted recovery unaided, and a co-intelligence mode, where participants used LMM advisory instructions. The results show that LMM assistance improved stuck recovery rates by 70% compared to unaided and unskilled human driving.