A Critical Epistemological Audit of Early Evidence Syntheses in Generative AI Educational Research: A Systematic Review and Meta-Analysis

Abstract

The rapid emergence of ChatGPT in education was accompanied by an equally rapid proliferation of meta-analyses synthesizing its effects on educational outcomes. We conducted a systematic epistemological audit of the first wave of these meta-analyses, evaluating both their methodological and reporting quality and the primary evidence they synthesized. We included meta-analyses published after the release of ChatGPT that quantitatively synthesized its effects on educational outcomes. Five meta-analyses comprising 87 unique primary studies met our inclusion criteria. We also reconstructed their quantitative evidence base and conducted exploratory three-level random-effects meta-analyses to examine effect magnitude, heterogeneity, and robustness, and how these varied with theoretically and methodologically relevant characteristics. The five meta-analyses generally searched multiple databases, specified eligibility criteria, and examined heterogeneity and potential publication bias, but showed limitations in independent screening and coding, preregistration, open data and code, and transparent reporting. The underlying primary evidence likewise showed methodological and reporting limitations and substantial conceptual heterogeneity. Among 69 classifiable primary studies, 79.7% assessed performance exclusively, whereas only 11.6% assessed learning exclusively. The reconstructed meta-analysis yielded a positive average effect (g = 0.83, 95% CI [0.68, 0.97]), but heterogeneity was substantial (I² = 94.06%; 95% prediction interval [−0.75, 2.41]), and estimates were markedly attenuated in sensitivity analyses addressing small-study and selection effects. Although positive effects remained among stronger studies assessing learning, increasingly stringent quality criteria rapidly reduced the evidence base, limiting the certainty and generalizability of conclusions about learning effects. Overall, our findings caution against interpreting a single pooled estimate as a stable general effect of ChatGPT on learning and highlight the need for more rigorous, conceptually precise, transparent, and reproducible research.