Agentic RAG for Legal Question Answering in Civil Law: Evidence From the Korean Bar Examination

Abstract

In civil law systems, legal professionals navigate sources of law hierarchically, searching for statutes, looking up specific articles, finding relevant cases, and examining full judgment texts in an iterative process. We present an agentic retrieval-augmented generation (RAG) architecture that mirrors this exploration workflow, enabling large language models (LLMs) to autonomously select and iteratively invoke five specialized legal tools implemented via the Model Context Protocol (MCP). On 300 multiple-choice questions from the 2025–2026 Korean Bar Examinations, our system achieved 95.33% accuracy with GPT-5.1 and 92.67% with Claude Sonnet 4.5, outperforming both Closed Book and Naïve RAG baselines, with the gains over Naïve RAG statistically significant (McNemar’s test, $p \lt .001$ ). However, effectiveness proved model-dependent: Gemini 2.5 Pro scored below its Naïve RAG baseline despite identical tool access. To explain this divergence, we analyzed tool-use behaviors and identified three distinct patterns: Intensive Tool Use (GPT), Efficient Utilization (Claude), and Tool Aversion (Gemini). GPT achieved high accuracy with broad, iterative case searches; Claude reached comparable performance with fewer searches, gaining accuracy through extended thinking rather than additional retrieval; and Gemini frequently avoided tool use altogether. An ablation study further revealed that case law tools were associated with the largest accuracy gains, followed by statute tools and full judgment text access. The magnitude of each contribution varied across models. These findings indicate that realizing the benefits of agentic RAG depends on selecting models with sufficient tool-use propensity, as tool access alone did not guarantee performance gains in our experiments.