Edge-Oriented Lightweight YOLO-World with Cross-Modal Fusion Adaptation for Multimodal Object Detection and Deployment

Abstract

Multimodal detection models enable flexible object detection through text prompts, but YOLO-World-style models still incur high computational and storage costs on edge devices. To address this problem, this paper develops an edge-oriented adaptation framework for replacing the original visual encoder of YOLOv8l-worldv2 with a compact visual branch. The framework jointly considers visual-branch compression, multi-scale interface consistency, cross-modal feature compatibility, and edge-side inference, rather than optimizing these aspects independently. Specifically, a YOLOv7-based visual branch is reconstructed using depthwise separable convolutions, enhanced by DyHead, and compressed through sensitivity-guided Filter Pruning via Geometric Median (FPGM) under multi-scale interface constraints. An identity-initialized semantic adaptation layer and multi-template text prototypes are then introduced to alleviate the feature distribution mismatch between the compressed visual branch and the original cross-modal fusion space. Finally, offline text prototype generation and TensorRT-based INT8/FP16 mixed-precision inference are used for edge deployment. Experiments on BDD100K show that the lightweight visual encoder achieves 62.45% mAP@0.5 with 22.5 M parameters and 52.8 GFLOPs. After multimodal integration, the proposed model achieves 64.9% mAP@0.5 with 31.5 M parameters and 67.2 GFLOPs, reducing GFLOPs by 67.1% compared with YOLOv8l-worldv2 while causing only a 1.6 percentage-point accuracy drop. On the Jetson Orin Nano Super, the deployed model reaches 32.6 FPS with a model size of 32 MB, demonstrating its feasibility for edge perception scenarios.