Hierarchical Domain Adaptation with Multi-Level Attention for RGB-Thermal Object Detection

Abstract

Cross-modal domain adaptation for RGB-to-Thermal object detection (DAOD) remains a challenging task due to the large modality gap, feature distribution mismatch, and limited supervision on the thermal domain. Conventional domain adaptation detectors often fail to capture consistent semantic representations across modalities, resulting in unstable adversarial learning and suboptimal detection accuracy. To address these challenges, we propose HDAM (Hierarchical Domain Adaptation with Multi-Level Attention), an enhanced domain-adaptive detection framework built upon Faster R-CNN. HDAM progressively aligns cross-domain features from low- to high-level representations through a hierarchical attention mechanism. Specifically, it comprises three key components: (1) Entropy-Guided Cross-level Attention (ECA), which leverages discriminator-derived entropy responses to reweight pixel- and mid-level features for more stable cross-modal encoding; (2) Multi-branch Global Semantics and Discriminator (MGSD), which performs global adversarial alignment and integrates multi-scale spatial information through attention-enhanced branches, while introducing an auxiliary image-level semantic consistency loss; and (3) Instance-Context Alignment (ICA), which fuses ROI-level and contextual embeddings and applies dual instance-level adversarial losses for precise fine-grained adaptation. Extensive experiments on benchmark RGB-Thermal datasets demonstrate that HDAM effectively bridges the modality gap and achieves improved cross-domain detection performance compared to recent DAOD methods.