A Systematic Comparison of RAG Architectures for Geographic POI Question Answering Using OpenStreetMap Data
Abstract
Abstract. Retrieval-Augmented Generation (RAG) grounds Large Language Models in external knowledge, yet geospatial question answering presents a distinctive challenge: spatial relationships such as distance and direction are directly computable from coordinate data, blurring the role of vector or graph-based retrieval that prevails in general-domain RAG. We systematically compare five enhanced RAG architectures—Structured, GraphRAG, Hybrid, Adaptive, and Agentic—for geographic Point of Interest (POI) question answering, all built on a shared vector-retrieval substrate over 1,047 OpenStreetMap POIs in Shibuya, Tokyo, with multiarea generalization tested across four Tokyo districts (about 3,600 POIs). Evaluation employs a hierarchical five-level prompt framework (L1–L5, 90–130 cases per phase) with multi-dimensional scoring covering keyword success, reasoning quality, evidence citation, constraint satisfaction, and uncertainty acknowledgement. In Phase 1 (90 cases), Structured RAG attained 89.1% versus GraphRAG’s 76.7% and Adaptive RAG’s 86.1% (Wilcoxon, Bonferroni-corrected, p < 0.001); per-category analysis identified two query types (directional comparison, competitor density) where GraphRAG remained superior. In Phase 2 (130 cases, four areas), Hybrid RAG achieved the best balance of composite quality (67.1/100) and cross-level stability, though pairwise differences with Adaptive and Graph RAG were not statistically significant. Findings suggest that, in dense-urban POI settings where coordinates are reliable, the marginal benefit of explicit graph edges shrinks for coordinate-computable relationships, while structured spatial processing complements vector retrieval. All software (ChromaDB, NetworkX, Hugging Face Transformers) and data (OpenStreetMap) are open-source, ensuring FOSS4G-community reproducibility.