VizRAG is introduced, the first RAG system to support visual hypergraph structure awareness, and demonstrates that VizRAG significantly outperforms strong baselines, validating the promising potential of hypergraph visualization as a novel approach for RAG systems.
Abstract
Hypergraph-based RAG systems surpass traditional graph-based approaches by organizing complex n-ary atomic facts among entities, rather than relying solely on binary relationships. Despite the advancements in multimodal large language models (MLLMs) with enhanced visual capabilities, current hypergraph-based RAG frameworks predominantly restrict knowledge retrieval and reconstruction to a unimodal, text-centric paradigm. This limitation prevents them from fully leveraging the powerful visual perception capabilities of modern MLLMs. To address this gap, we systematically explore the integration of hypergraph awareness in RAG systems through visual cues. By incorporating visual representations of hypergraphs into the RAG pipeline, we introduce VizRAG, the first RAG system to support visual hypergraph structure awareness. Experimental results demonstrate that VizRAG significantly outperforms strong baselines, validating the promising potential of hypergraph visualization as a novel approach for RAG systems.
This paper formalizes the document structure as a Multimodal Hypergraph, utilizing hyperedges as unified semantic containers to encapsulate multi-way associations across text, images, and tables, thereby transcending point-to-point modeling and introducing an Anchor-driven Incremental Refinement mechanism.
Shenao Chen, Yidan Xu, Xiangmin Han et al.· 0 citations
Retrieval-Augmented Visually-rich Generation (RAViG) extends RAG by integrating textual explanations with multiple visual elements in a well-structured layout. Despite its growing adoption, no existing benchmark offers a holistic evaluation of RAViG. Current RAG benchmarks focus on text-only generation, while natural l...
Qi-Rui Hu, Shunlei Ning, Chong Bao et al.· Proceedings of the 32nd ACM...· 0 citations
Retrieval-Augmented Generation (RAG) has become a fundamental paradigm for enhancing Large Language Models (LLMs) with external knowledge. However, while recent structure-augmented approaches organize documents into graphs to improve information access, their retrieval strategies remain largely static, relying on simil...
Jinghong Lei, Wang Kun, Zhigang Chen et al.· Proceedings of the 32nd ACM...· 0 citations
Retrieval-augmented generation (RAG) systems have traditionally relied on text-based pipelines that extract and retrieve information from documents. While efficient and lightweight, these approaches often struggle with documents where meaning is conveyed through layout, tables, and visual elements. Recent advances in m...
Retrieval-augmented generation (RAG) is widely used to mitigate hallucination issues in large language models (LLMs) and multimodal large language models (MLLMs). In particular, knowledge graph (KG)-based RAG leverages structured knowledge to provide (M)LLMs with high-quality external information. Building on these wor...
Zongyu Wu, Yilong Wang, Xiaochen Wang et al.· 0 citations
This tutorial provides an in-depth treatment of modern RAG based on AI-facilitated systematic analysis of ~2000 recent papers (2020--2026) and traces the RAG pipeline from its modular foundations through graph-enhanced reasoning to the latest RL-driven agentic architectures, covering each stage.
Xin Dong, Sanat Sharma, Kai Sun et al.· Proceedings of the 32nd ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
