Sep 2026· Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track· Vol abs/2609.35461· 0 citations· 18 references
Computer Science
TL;DR
AraDynFact is introduced, a novel dynamic evaluation framework designed to rigorously assess the factual Arabic knowledge embedded in LLMs, and employs a dynamic approach to extract factual information and generate rich and answerable questions in a fast and automatic way.
Abstract
As Large Language Models (LLMs) continue to scale both in size and capabilities, their proficiency in the Arabic Language has seen significant advancement. However, a critical gap remains: the extent of their factual knowledge and cultural sensitivity to the diverse Arabic-speaking world remains largely underexplored. Current evaluation metrics often focus on translation or generic reasoning, failing to capture the rich historical, social, and regional nuances inherent to Arabic culture. In addition, most benchmarks rely on heavy work, with human intervention in some steps, making the evaluation of knowledge coverage expensive and slow. To address this deficiency, we introduce AraDynFact, a novel dynamic evaluation framework designed to rigorously assess the factual Arabic knowledge embedded in LLMs. Unlike static benchmarks, AraDynFact employs a dynamic approach to extract factual information and generate rich and answerable questions in a fast and automatic way. We apply AraDynFact to Arabic Wikipedia and audit the performance of several state-of-the-art models, ranging from Arabic-centric specialized LLMs to high-resource general purpose LLMs. In addition we found a high degree of correlation with existing, hand-crafted Arabic-centric benchmarks, confirming the potential of our dynamic approach.
It is examined how Arabic’s key linguistic features—non‑concatenative morphology, broken plurals, diacritic ambiguity, and root‑based semantics—affect enrichment outcomes and assess their effects across different enrichment techniques.
Leila Nouri, H. Aliane· ACM Transactions on Asian an...· 0 citations
Multilingual benchmarks for African languages mostly translate generic questions into local languages, measuring linguistic competence but not whether models know facts about Africa. We introduce \naija, a benchmark of factual and statistical questions about Nigeria, authored by native speakers and grounded in authorit...
Zainab Afolabi, Nikita Kozodoi, Jack Butler et al.· 0 citations
Arafa is introduced, a new large-scale dataset for fact-checking in Modern Standard Arabic, constructed through an automated framework leveraging large language models (LLMs) and presented a scalable approach for developing similar resources for other low-resource languages.
Christophe Khalil, Shady Elbassuoni, Rida Assaf· arXiv.org· 0 citations
Figurative language remains a major challenge for Machine Translation (MT) systems, as the intended meaning of such expressions often goes beyond literal translation. Although recent advances in Large Language Models (LLMs) improved the translation of Arabic dialects, benchmark resources for evaluating figurative langu...
Jihad R’baiti, Youssef Hmamouche, E. Chersoni et al.· 0 citations
E-CONAN diagnostics dataset is introduced, a freely available dataset manually-annotated with coarse-grained and fine-grained categories based on the authors' proposed Arabic hierarchy, which helps NLU designers better understand their models by doing error analysis and in-depth investigation.
Khloud Al Jallad, Nada Ghneim, Ghaida Rebdawi· arXiv.org· 0 citations
Senior Research Scientist Jim Glass discusses how his MIT-IBM Computing Research Lab collaboration connects perception and language, and addresses the shortcoming of current models.
MIT News · Artificial Intelligence· news.mit.eduNov 19, 2026
An MIT and Dana-Farber Institute-led team used large language models to examine nearly 4,000 studies and hundreds of code repositories, revealing gaps in the information needed to scrutinize and reuse published research.
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.