Research on Multimodal UAV Object Tracking Based on a Dual-Branch Interactive Fusion Encoder
Abstract
: Unmanned Aerial Vehicle (UAV) target tracking is essential for security surveillance and search-and-rescue missions. However, existing methods primarily rely on single-modal visible light imagery, leading to high missed-detection rates and poor robustness in complex environments such as nighttime, fog, and occlusion. While multimodal fusion offers mitigation, current approaches are constrained by coarse cross-modal feature alignment, high computational complexity, and challenges in suppressing modality bias. To address these issues, this study proposes a novel multimodal UAV target tracking framework integrating a Dual-Branch Interactive Multimodal Detection Network (DBIMDet) with the ByteTrack tracker. Built upon the Real-Time Detection Transformer (RT-DETR), the network employs parameter-decoupled Residual Network-18 backbones to extract multi-scale visible and infrared features. It introduces a Cross-Modal Interactive Fusion Encoder (CMIFE) utilizing Polarized Linear Attention for efficient global modeling, and incorporates a Spatially-Enhanced Feed-forward Network (SEFN) to inject spatial priors. Additionally, Modality-Aware Bypass Adaptive (Mona) and Dynamic Hyperbolic Tangent (DyT) normalization are employed to mitigate modality bias, while a Multimodal Collaborative Loss (MCLoss) guides the enhancement of modality consistency and complementarity. Extensive experiments on the DroneVehicle dataset demonstrate that, among the compared methods evaluated in this study (MM-Det, RSDet, UA-CMDET, and IVD-NET), the proposed method achieves the best performance on this benchmark, with an mAP50 of 84.33% and an mAP50-95 of 60.26%. Real-world nighttime video evaluations further validate its adaptability under the evaluated challenging illumination conditions. The framework provides a robust solution through efficient fusion and bias suppression, ensuring reliable UAV tracking in challenging conditions.