Improving Groundedness in Policy Document Question Answering Using Hybrid RAG with Structure-Based Topic Chunking : A DTNN 2030 Case Study

Abstract

This paper presents a Hybrid Retrieval-Augmented Generation (RAG) framework for question answering on long-form policy documents. Dense-only retrieval, a common baseline in RAG pipelines, may struggle with exact lexical matching for structured information such as technical terminology, acronyms, and domain-specific entities commonly found in policy documents. To address this limitation, this study proposes a structure-based topic chunking method that enriches retrieval signals using three information layers: a keyword layer for exact lexical matching, a contextual summary layer for preserving higher-level document meaning, and a content layer containing the original raw text. The proposed retrieval pipeline combines FAISS dense retrieval, BM25 sparse retrieval, Reciprocal Rank Fusion (RRF), and cross-encoder reranking to improve retrieval quality and contextual relevance. During answer generation, only the original content layer is forwarded to the Large Language Model (LLM), allowing the effect of retrieval quality on response groundedness to be evaluated independently. The framework was evaluated on the Dasar Teknologi Nuklear Negara (DTNN) 2030, a Malaysian national policy document on nuclear technology development, using 32 manually constructed questions across five question categories. Experimental results show that the Hybrid + Reranker approach achieved an overall score of 126/192 (65.6%) compared to 71/192 (37.0%) for dense-only retrieval, representing an improvement of 28.6 percentage points. The largest improvement was observed in groundedness, where ungrounded responses were reduced from 53.1% to 3.1%. These findings suggest that, in this case study, the retrieval strategy was the main factor behind the improved groundedness on DTNN 2030.