Vision-Language Model-Guided Transparent Object Perception and Task-Oriented Grasping for Robotic Manipulation

Abstract

Transparent objects such as glass containers, test tubes, and plastic bottles are common in robotic manipulation scenarios, but their refractive and reflective surfaces produce incomplete RGB-D geometry and make task-specific grasp selection unreliable. This paper presents an integrated vision-language system for transparent object perception and task-oriented grasping. First, we construct VLM-DRE, a transparent object image instruction dataset with 12,700 images and 38,100 image-instruction-bounding-box triplets. LoRA fine-tuning of Molmo-7B improves target click accuracy from 86.4% to 91.5% and IoU@0.75 from 57.5% to 69.1%. Second, MSR-Net performs monocular depth completion and mask prediction using multi-scale adaptive feature fusion and progressive feature refinement, achieving RMSE 0.066, mAP 98.61%, and IoU 94.12% on Syn-TODD, and RMSE 0.118, mAP 99.02%, and IoU 87.95% on ClearPose. Third, LMF-Net combines RGB-D cross-modal fusion with learnable multi-factor matching to rank AnyGrasp 6-DoF candidates, reaching 77.8% Top-1 and 90.5% Top-3 accuracy on TaskGrasp-Image and improving PRISM-Real success from 61.1% to 68.5%. On a RealSense D435i–Unitree Z1 Pro platform, the complete system obtains 85.4% success with manual clicks and 71.3% with VLM-predicted clicks, supporting perception-to-grasping integration while highlighting target localisation and runtime as deployment bottlenecks.