Legalyze: An Open‐Source Structure‐Aware Agentic RAG Platform for High‐Precision Legal Intelligence

Abstract

The integration of Large Language Models into the legal domain is constrained by a reliability gap, where models can generate incorrect or unsupported information in specialized reasoning tasks. This issue is especially important in jurisdictions such as Pakistan, where legal texts have strong structural dependencies and require accurate grounding. This research introduces Legalyze, an open‐source and modular five‐tier system designed to support legal analysis. The system uses Structure‐Aware Hierarchical Indexing (SAHI) and a dual‐stream retrieval pipeline that combines lexical and semantic search using Reciprocal Rank Fusion. It also includes a component called Legal Wisdom of Extraction, which identifies key legal entities such as petitioners, respondents, court names and referenced laws from the text. By preserving the structure of legal documents and grounding responses in retrieved evidence, the system substantially improves the grounding of generated outputs. Evaluation on a 1.2 million‐token corpus of Pakistani statutes and case law, comprising five statutory and case‐law sources with 67 legal queries assessed by three independent legal reviewers under a reviewer‐blinded evaluation protocol (Fleiss' κ=0.743$$ \kappa =0.743 $$ , substantial agreement), showing a reduction in pooled hallucination rates from 54.7% in a zero‐shot baseline to 5.5%, outperforming standard retrieval‐based approaches. The improvement over both baselines is statistically significant across all pairwise comparisons (Cochran's Q=37.57$$ Q=37.57 $$ , p=6.93×10−9$$ p=6.93\times {10}^{-9} $$ ; McNemar exact: Zero‐Shot vs. Legalyze p=5.40×10−9$$ p=5.40\times {10}^{-9} $$ ). A seven‐configuration offline retrieval ablation further demonstrates that an offline surrogate of the Legalyze architecture achieves Recall@5 of 65.0%, compared to 45.0% for flat lexical retrieval, confirming the retrieval gains attributable to structure‐aware indexing. These results show that structure‐aware retrieval combined with targeted information extraction can improve the reliability of AI systems for legal research and analysis. Legalyze represents a promising prototype for delivering trustworthy, jurisdiction‐specific legal intelligence as a research and decision‐support tool in professional environments.