Skip to content
#natural language processing Conference Open access

AraDynFact: Dynamic Evaluation of Factual Knowledge in Arabic

Sep 2026 · Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track · Vol abs/2609.35461 · 0 citations · 18 references
Computer Science

TL;DR

AraDynFact is introduced, a novel dynamic evaluation framework designed to rigorously assess the factual Arabic knowledge embedded in LLMs, and employs a dynamic approach to extract factual information and generate rich and answerable questions in a fast and automatic way.

Abstract

As Large Language Models (LLMs) continue to scale both in size and capabilities, their proficiency in the Arabic Language has seen significant advancement. However, a critical gap remains: the extent of their factual knowledge and cultural sensitivity to the diverse Arabic-speaking world remains largely underexplored. Current evaluation metrics often focus on translation or generic reasoning, failing to capture the rich historical, social, and regional nuances inherent to Arabic culture. In addition, most benchmarks rely on heavy work, with human intervention in some steps, making the evaluation of knowledge coverage expensive and slow. To address this deficiency, we introduce AraDynFact, a novel dynamic evaluation framework designed to rigorously assess the factual Arabic knowledge embedded in LLMs. Unlike static benchmarks, AraDynFact employs a dynamic approach to extract factual information and generate rich and answerable questions in a fast and automatic way. We apply AraDynFact to Arabic Wikipedia and audit the performance of several state-of-the-art models, ranging from Arabic-centric specialized LLMs to high-resource general purpose LLMs. In addition we found a high degree of correlation with existing, hand-crafted Arabic-centric benchmarks, confirming the potential of our dynamic approach.

Read PDF

Similar papers

Review Open access Oct 2026

Arabic WordNet Enrichment: A Comprehensive Survey

It is examined how Arabic’s key linguistic features—non‑concatenative morphology, broken plurals, diacritic ambiguity, and root‑based semantics—affect enrichment outcomes and assess their effects across different enrichment techniques.

Leila Nouri, H. Aliane · 0 citations
#artificial intelligence Preprint Nov 2026

NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria

Multilingual benchmarks for African languages mostly translate generic questions into local languages, measuring linguistic competence but not whether models know facts about Africa. We introduce \naija, a benchmark of factual and statistical questions about Nigeria, authored by native speakers and grounded in authorit...

Zainab Afolabi, Nikita Kozodoi, Jack Butler et al. · 0 citations

ARAFA: An LLM-Generated Arabic Fact-Checking Dataset

Arafa is introduced, a new large-scale dataset for fact-checking in Modern Standard Arabic, constructed through an automated framework leveraging large language models (LLMs) and presented a scalable approach for developing similar resources for other low-resource languages.

Christophe Khalil, Shady Elbassuoni, Rida Assaf · 0 citations
Preprint Oct 2026

Towards Figurative Language Understanding in Moroccan Darija: An Evaluation of Large Language Models

Figurative language remains a major challenge for Machine Translation (MT) systems, as the intended meaning of such expressions often goes beyond literal translation. Although recent advances in Large Language Models (LLMs) improved the translation of Arabic dialects, benchmark resources for evaluating figurative langu...

Jihad R’baiti, Youssef Hmamouche, E. Chersoni et al. · 0 citations

E-CONAN (Entailment, CONtradition And Neutral) Diagnostics Dataset Investigating Linguistic Phenomena in Arabic Natural Language Understanding

E-CONAN diagnostics dataset is introduced, a freely available dataset manually-annotated with coarse-grained and fine-grained categories based on the authors' proposed Arabic hierarchy, which helps NLU designers better understand their models by doing error analysis and in-depth investigation.

Khloud Al Jallad, Nada Ghneim, Ghaida Rebdawi · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.