Beyond Lexical Overlap: A Multi-Dimensional Evaluation Framework for Arabic Retrieval-Augmented Generation Systems

Abstract

Retrieval-Augmented Generation (RAG) has become a standard technique to ground large language model outputs in external knowledge. However, evaluating RAG systems for Arabic remains problematic because traditional lexical metrics such as ROUGE and BLEU were designed for English, a language with limited morphological variation. Arabic's rich derivational system where a single triliteral root can produce dozens of surface forms causes these metrics to penalize correct paraphrasing while being completely bling to factual hallucination. This paper introduces a holistic evaluation framework that combines hybrid retrieval (BM25 with dense embeddings from BGE-M3) and a multi-dimensional scoring suite. We compare six LLMs three Arabic specialized (ALLaM-7B, Fanar-1-9B, Noon-7b) and three multilingual (Llama-3.3-70B, Qwen-2.5-7B, BLOOM-7B) across 300 queries over a general Arabic corpus of 30 documents. Our hybrid retriever achieves aRecall@5 of 0.942 and an MRR of0.883. The central empirical finding is a metric mirage: the Pearson correlation between ROUGE-1 and semantic similarity (measured by BGE-M3) is only 0.317, meaning that more than 90% of factual variance is invisible to lexical overlap metrics. Llama-3.3-70B achieves the highest composite score (2.703/4), driven by superior named entity recognition accuracy (0.463). However, the smaller Arabic-specialized ALLaM-7B exhibits strong performance across several of evaluation metrics—particularly named entity recognition (NER), ROUGE-1, ROUGE-2, and BLEU. https://github.com/AAA20121/Beyond-Lexical-Overlap.git.