VLM-guided retrieval augmented generation (RAG) for robot action prediction

Abstract

Reliable action prediction is essential for robotic electronic-waste disassembly, where device diversity and damage make preplanned sequences impractical. Vision-language models (VLMs) offer broad visual and semantic knowledge, but zero-shot predictions can be unreliable, while fine-tuning requires substantial task-specific data. We propose a VLM-guided retrieval-augmented generation (RAG) framework that crops task-relevant image regions, retrieves examples from a structured local database, re-ranks candidates, and expands the search when needed. We evaluated the framework on six types of smoke detectors and heat-cost allocators and compared the performance of three VLMs using zero-shot inference, single-example RAG, VLM-guided RAG, and task-specific fine-tuning. Cropping improved retrieval across all evaluated models, while VLM-guided re-ranking increased top-1 retrieval accuracy from 68.47% to 84.57-89.29%. Next action prediction accuracy reached 90.82-95.92% and significantly outperformed single-example RAG. RAG-based methods also outperformed the fine-tuned models. Median prediction latency was 8.95 s, although expanded searches were slower. Overall, VLM-guided RAG enables accurate, data-efficient adaptation to new device types without fine-tuning, at the cost of increased and variable latency.