HRU: Harmonizing Reading and Understanding for Parameter-Efficient Scene-Text Aware Cross-Modal Retrieval

Abstract

Scene-text aware cross-modal retrieval requires models to jointly capture visual text content and object semantics. However, existing methods often exhibit an inherent imbalance: they either over-rely on fine-grained textual cues and become sensitive to noise, or overlook critical textual information, resulting in suboptimal alignment. To address this issue, we propose HRU, a parameter-efficient framework that Harmonizes Reading and Understanding via a two-stage progressive learning strategy and orthogonality constraints. In the first stage, visual prompts disentangle object-level and text-level representations, enabling adaptive scene text encoding while preserving visual semantics. In the second stage, a fusion mechanism progressively aligns and refines these features, achieving coherent heterogeneous visual information integration. With a parameter-efficient design, HRU can be seamlessly adapted to multimodal backbones of varying scales with minimal fine-tuning overhead. Extensive experiments demonstrate that HRU significantly improves alignment and retrieval robustness, achieving RSUM scores of 474.5, 380.3, and 546.3 on CTC-1k, CTC-5k, and TextCaps, respectively, with fewer than 30K additional trainable parameters. These results establish HRU as an effective and unified solution for text-rich visual understanding and retrieval.