The novel task of cross-modal query suggestion is introduced, which interactively guides users by suggesting textual refinements based on visual clusters identified in the retrieval results, and the creation of CroQS, a benchmark dataset comprising 50 diverse queries and 295 semantic clusters in generic domain.
Text-to-image retrieval (T2IR) aims to retrieve semantically relevant images given natural language queries. However, existing systems typically adopt static retrieval pipelines that process all queries uniformly, despite differences in the informativeness of individual query tokens. To address this limitation, we propose a retrieval-time token-aware query reformulation method that emphasizes informative query tokens without retraining or changes to the retriever architecture. Our method uses token-level routing to determine when query adaptation should be applied and constructs a focused query embedding by pooling salient tokens while preserving global semantic context. The reformulated query representation is used for first-stage retrieval, and the retrieved candidates are subsequently reranked. Experiments on MS COCO and Flickr30k show modest overall gains over the corresponding base pipeline, while also providing interpretable token-level evidence and region-level visualizations. Overall, our results show that retrieval-time token-aware query reformulation improves existing T2IR pipelines while making the role of individual query tokens more explicit.
Seonah Kim, Minkeon Kim, Youjin Lee et al.· International Conference on...· 0 citations
Composed image retrieval (CIR) seeks to retrieve target images using multi-modal queries, specifically a reference image paired with modification text. Central to CIR is integrating textual semantic modifications with visual content. Despite its importance, existing approaches typically employ a static fusion paradigm, failing to account for the semantic heterogeneity of user queries, which encompass diverse task types (e.g., addition, replacement) and var-ied content. To address these limitations, we propose the Task-Adaptive Hier-archical Prompt (TAHP) framework. TAHP guides feature extraction through dynamically generated, task-specific prompts structured at three hierarchical levels: task-type, task-content, and general prompts. Furthermore, we design a Prompt Dynamic Generation Module to adaptively synthesize prompts condi-tioned on user queries and introduce a False Negative Correction Loss to optimize cross-modal feature fusion. Extensive experiments on FashionIQ and CIRR datasets demonstrate that TAHP achieves state-of-the-art performance against existing CIR approaches.
Zeli Yan· Poster Volume 0008 The 2026...· 0 citations
In domain-specific multimodal long documents, images and text jointly convey complex knowledge that cannot be fully captured by plain text alone. However, existing paradigms like DocVQA primarily focus on generating textual answers or localizing evidence regions, rather than outputting query-specific textual attribute values and corresponding images. To address this gap, we propose query-driven image-text joint extraction from long documents, requiring models to output query-requested textual attribute values and corresponding image bounding boxes. Based on challenges related to both user intent and document content, we designed a two-level taxonomy that operates at the query and instance levels. Further, we construct ITJoint, the first high-quality, manually annotated benchmark for this new task, comprising 2,455 pages of domain-specific documents with numerous non-decorative images, 316 queries, and 910 answer instances. Finally, we evaluate representative standalone Vision-Language Models from different providers and further design Q2IT, a multi-agent collaborative framework consisting of three progressively collaborating agents for evidence collection, page selection, and target-image localization. Using a joint evaluation approach that assesses both text extraction and image localization, our experiments show that standalone VLMs struggle with this task, while Q2IT significantly improves performance on ITJoint, although a substantial gap remains toward perfect results.
Recent page-image retrievers such as ColPali have improved retrieval over visually rich documents, yet little is known about how they behave in cross-lingual, low-resource settings. We introduce L\"etzCross, a benchmark for cross-lingual page-level retrieval over Luxembourgish PDF documents, with document pages indexed as images and queries provided in English, French, German, and Luxembourgish. The benchmark combines text-focused QA pairs with visually grounded QA pairs, covering both textual and visual retrieval needs in PDF-based RAG. We use L\"etzCross to compare OCR-based text-only retrievers with ColPali-style page-image retrievers and find that the latter perform better across query languages in this system-level comparison. We also examine single-language and multilingual fine-tuning. Fine-tuning transfers across query languages, with French yielding the highest mean performance on Luxembourgish queries among the single-language settings. In the multilingual setting, including Luxembourgish gives the strongest results and substantially improves retrieval for Luxembourgish queries.
Omar El Bachyr, Fred Philippy, Laura Bernardy et al.· 0 citations
Given a reference image, composed image retrieval aims to search for the target image from the image gallery with a modification sentence. The complicated semantic relationships within the composed query to comprehend the preserved visual context and modified semantics pose a challenge for the researchers. Most current approaches concentrate on the global or fragmental composition to combine the multi-modal query features and bridge the distance between the query and the target. However, global-level matching tends to ignore fine-grained components that distinguish the matched instance from hard negative instances. As for the fragment-level matching, the over-sensitivity to the local noise due to the lack of explicit local labels and excessive computational demands may have a detrimental impact on performance. In this paper, we propose Semantic Composition via Optimal Transport (SCOT) for composed image retrieval to address the aforementioned issues. To optimize the computational resources, SCOT first introduces Hybrid-modal Affinity Summarization (HAS) to compact salient relationships in the affinity matrix from the reference image patches and modification tokens. Furthermore, to mitigate the over-confidence in local noise, SCOT adaptively assigns optimal transport weights on the salient aligned fragments to Retain Prototype from Alignments (RAS) and guides discriminant semantics learning to Update Semantics from Enhanced Embeddings (USEE), while preserving the characteristics from global structures. Specifically, RAS module investigates the preserved semantics from the reference image based on the affinity matrix, meanwhile USEE module distills significant modification semantics. The proposed SCOT has demonstrated advanced experimental performances on widely-used datasets FashionIQ, CIRR, and Fashion200K. Ablative studies manifest the effectiveness of optimal transport guidance on preserved visual prototypes and updated semantics.
Yifan Wang, Wuliang Huang, Chun Yuan· IEEE Transactions on Image P...· 0 citations
Composed Image Retrieval (CIR) is a popular multi-modal retrieval task that aims to retrieve a target image based on a query composed of a reference image and modification text. The challenge lies in how to effectively retrieve a target image that preserves the visual content of the reference image while incorporating the changes described by the modification text. Existing CIR methods primarily employ a fusion-based strategy or a textual-inversion strategy during training. Although these methods have achieved promising results, they are limited in fully leveraging multi-modal information. This results in modality redundancy, where the retrieval process is dominated by one modality while ignoring the other. To address this issue, we propose an asymmetric fusion mechanism to generate dual retrieval queries of different granularity, enabling the model to fully use multi-modal information. Specifically, we propose a novel method termed Dual Retrieval Queries Fine-Tuning for Composed Image Retrieval (DRQ-CIR), which consists of two key components: 1) a Bilateral Multi-Modal Fusion (BMMF) module based on pre-trained VLMs, which combines the reference image and modification text to generate an enriched retrieval query; and 2) a Dual Retrieval Queries Fine-Tuning (DRQ-FT) module, which employs latent prompts to generate an enhanced retrieval query. Dual retrieval queries are used for contrastive learning with the target image to fine-tune the model and improve retrieval performance. Additionally, we introduce a bi-directional training paradigm to ensure retrieval consistency and further exploit the triplets. Extensive experiments validate the effectiveness of our proposed method on four established benchmark datasets. (Code will be available at: https://github.com/Crystal-twy998/DRQ-CIR).