Skip to content
Book Open access

Rethinking Graph-LLMs: Are Performance Gains Confounded by Data Leakage?

Nov 2026 · Proceedings of the 35th ACM International Conference on Information and Knowledge Management · 0 citations · 36 references

Abstract

Large Language Models (LLMs) have advanced text-attributed graph (TAG) learning in recent years, leading to strong performance in Graph-LLM systems. However, these gains may partially arise from target-correlated parametric priors memorized during LLM pre-training rather than genuine graph reasoning. This issue is particularly critical in TAGs, where graph topology can propagate memory-triggered signals beyond individual nodes. Once encoded as node representations, memorized priors may spread through neighborhood aggregation, introducing topology-level bias amplification absent in conventional NLP tasks. Despite its importance, the interaction between LLM memorization and graph message passing remains largely unexplored. In this paper, we present a leakage analysis framework guided by Structural Causal Models (SCMs) for Graph-LLMs. Specifically, we investigate how prompt interventions influence downstream graph prediction and examine whether entity-level anonymization suppresses memory-triggering cues. We further construct globally consistent entity-anonymized graph benchmarks that preserve graph structure while masking identity-sensitive entities. Experiments on citation and person relational networks across multiple embedding models and graph backbones reveal three findings: (1) Target-related meta-information can act as shortcut signals when highly correlated with downstream labels; (2) Graph message passing may further propagate memory-induced bias across neighboring nodes; and (3) Graph-LLM performance drops notably after entity anonymization. These findings suggest that current Graph-LLM benchmarks may overestimate generalization ability when memorized priors are not properly controlled. The code and anonymized datasets are publicly available at: https://github.com/mrjc12/GL-DataLeakage.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.