Results support hypergraph-native traversal with temporal-spatial-aware scoring as an effective retrieval paradigm for the evaluated multimodal RAG setting, particularly for constraint-driven queries that semantic-only retrieval cannot address.
Abstract
Existing graph-based retrieval-augmented generation (RAG) systems represent knowledge with binary relations and rely primarily on semantic similarity for retrieval. This design struggles with multimodal queries requiring temporal constraints, spatial relationships, or higher-order interactions among entities. We present HyperTSRAG, a multimodal RAG retrieval algorithm that models knowledge as a hypergraph and performs explicit temporal-spatial reasoning during traversal. HyperTSRAG performs a bipartite-alternating best-first traversal that alternates between entity nodes and hyperedge nodes to capture higher-order connectivity and ranks candidate evidence using a scoring function applied within traversal that integrates semantic similarity, temporal coherence, spatial overlap, and structural importance. We evaluate HyperTSRAG on a benchmark corpus of 1,000 multimodal documents (text, images, audio, and video) with 500 queries spanning simple lookups through complex multi-hop reasoning. For complex queries, HyperTSRAG achieves 78.3% Recall@10, improving by 12.3% over GraphRAG and 18.3% over LightRAG, while maintaining a 95th-percentile latency of 1.83 s. On temporal-spatial subsets, HyperTSRAG attains 85.2% accuracy on temporal queries and 81.6% on spatial queries. Ablation studies show that individual scoring components contribute 2–5% gains. Relative to a matched Projected-Binary-RAG control derived from the same extracted hyperedges, HyperTSRAG improves Recall@10 by 9.1%, while a separate binary-graph traversal ablation yields a 13.4% decrease, underscoring the value of native n-ary structure. These results support hypergraph-native traversal with temporal-spatial-aware scoring as an effective retrieval paradigm for the evaluated multimodal RAG setting, particularly for constraint-driven queries that semantic-only retrieval cannot address.
This paper formalizes the document structure as a Multimodal Hypergraph, utilizing hyperedges as unified semantic containers to encapsulate multi-way associations across text, images, and tables, thereby transcending point-to-point modeling and introducing an Anchor-driven Incremental Refinement mechanism.
Shenao Chen, Yidan Xu, Xiangmin Han et al.· 0 citations
It is suggested that step-wise CoT reasoning can improve node selection quality in hierarchical traversal, with gains most evident on standard-difficulty questions, indicating that the quality of the generated reasoning signal is a key bottleneck for further improvement.
Retrieval-Augmented Generation (RAG) has become a fundamental paradigm for enhancing Large Language Models (LLMs) with external knowledge. However, while recent structure-augmented approaches organize documents into graphs to improve information access, their retrieval strategies remain largely static, relying on similarity ranking or static probability diffusion. We identify that this paradigm suffers from two inherent limitations in complex reasoning: popularity bias, where retrieval paths are trapped by high-degree distractors, and signal decay, where relevance signals attenuate over long reasoning chains. To overcome these challenges, we propose NaviRAG, a novel framework that reformulates retrieval as a reinforcement learning-driven dynamic navigation problem on schema-less knowledge graphs (KGs). Unlike passive diffusion, NaviRAG employs an agent that actively traverses the graph to act as a search-space pruning engine, identifying logical multi-hop reasoning paths. Technically, we introduce three key components: (1) Structure-Aware Query Expansion, which bridges the modality gap between unstructured queries and structured graph seeds for precise initialization; (2) Target-Driven Reward Shaping, which provides dense supervision based on semantic progress toward gold documents, effectively mitigating the sparse reward problem in large-scale graph traversal; and (3) a Multi-View Hybrid Reranking strategy that operates on the highly-pruned candidate subgraph, integrating policy confidence, semantic relevance, and global structural importance to ensure robust candidate selection. Extensive experiments on three multi-hop QA datasets and two single-hop QA datasets demonstrate that NaviRAG significantly outperforms baselines, achieving state-of-the-art performance in multi-hop QA while maintaining robustness in single-hop QA. Our code and data are available at https://github.com/CkingEW/NaviRAG.
Jinghong Lei, Wang Kun, Zhigang Chen et al.· Proceedings of the 32nd ACM...· 0 citations
Experimental results demonstrate that the VDGR-RAG method significantly outperforms a variety of RAG baselines in terms of both knowledge retrieval recall and QA accuracy.
Wenqi Chen, Haofei Yang, Rui Yang et al.· 0 citations
The Document Question Answering (DocQA) task necessitates the synergistic interpretation of visual and textual information embedded within documents. Although Retrieval-Augmented Generation (RAG) has enhanced the capabilities of Large Vision-Language Models (LVLMs), existing approaches still encounter significant bottlenecks when processing large-scale documents: the inability to capture long-range contextual dependencies within non-textual modalities, the difficulty in facilitating interaction and mutual complementation between different modalities, and the inefficient integration of heterogeneous modal information. To address these challenges, we introduce MMHRAG, a novel Multi-Modal Hierarchical Retrieval-Augmented Generation framework that achieves cross-modal interaction on DocQA for the first time. First, we construct a cross-modal hierarchical retrieval tree via a bottom-up recursive clustering and summarization mechanism. A key innovation of our approach is the structural injection of visual information, where image semantics are integrated as high-level abstract summaries of textual segments, thereby bridging the semantic gap between modalities. Furthermore, we design a Summarizing Agent to resolve logical conflicts, information redundancy, and granularity discrepancies among retrieved cross-modal evidence. Extensive experiments across multiple multi-modal long-document benchmarks demonstrate that MMHRAG significantly outperforms state-of-the-art baselines, achieving superior accuracy and consistency in complex reasoning tasks.
Jia-Yuan Wang, Jie Lian, Fu Zhao et al.· Proceedings of the 32nd ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
