Preprint

Why Does Graph Learning Fail to Fully Benefit from a Text Teacher?

Fumiaki Kimino Ryoma Sato Sokendai National Institute of Informatics

Aug 2026 · 0 citations · 35 references

Computer Science

TL;DR

A multimodal model that combines two complementary ideas: a self-supervised method that enables a GNN encoder pretrained on one dataset to operate directly on another dataset with a different node-feature dimensionality, without rebuilding the model or realigning the data is investigated.

Abstract

Graph neural networks (GNNs) are widely used to represent complex interactions and relationships among entities. We investigate a multimodal model that combines two complementary ideas: a self-supervised method that enables a GNN encoder pretrained on one dataset to operate directly on another dataset with a different node-feature dimensionality, without rebuilding the model or realigning the data; and an alternating optimization method that updates a language-model module in an E-step and a GNN module in an M-step, rather than jointly training a large language model and a GNN end to end on a large graph. Despite expectations, the combined model did not sufficiently improve predictive performance. We identify six factors: (1) an external anchor in the E-step has a strength-safety trade-off: a weak anchor has little effect, whereas an overly strong anchor can damage the graph representation; (2) the knowledge of the E-step teacher is not injected directly into the GCN embedding Z; (3) the representation space constructed in the M-step is not optimized for the same objective as the E-step teacher space, resulting in a compromise representation for target classification; (4) GCN propagation averages a node's own textual information with information from its neighbors; (5) cosine alignment does not guarantee axes that are discriminative for classification, so stronger geometric alignment with the E-step text anchor need not sufficiently improve the target decision boundary or classification performance; and (6) the force that preserves the source-side self-supervised geometry in the M-step conflicts with the force that moves the representation toward the E-step teacher. We support these observations through a staged set of experiments that varies the influence of the E-step.

arXiv Semantic Scholar

View source

Similar papers

#graph neural networks Aug 2026

Taking lessons from history: Memory Augmented Self Distillation for graph neural networks

This study introduces a novel memory-augmented self-learning framework that extracts and provides diverse learning sources for adaptive knowledge distillation from the student model itself, resulting in a 2.5-6% increase in accuracy across various benchmark datasets compared to current GNN training and self-distillation methods.

Saurabh Sharma, Souvik Chowdhury, Joydeep Chandra · Data mining and knowledge di... · 0 citations

#machine learning Preprint Aug 2026

Who Should Teach? Confidence-Aware Dual-Teacher Learning for Few-Shot Node Classification on Text-Attributed Graphs

This work proposes CoTeach, a Confidence-aware dual-teacher learning framework that dynamically selects the more reliable teacher for each node, and demonstrates that CoTeach consistently improves few-shot node classification performance while reducing unnecessary LLM utilization and associated monetary costs.

Hojin Kim, Sujin Yoon, Sung-Su Lim et al. · 0 citations

#machine learning Preprint Sep 2026

Self-Supervised Pretraining of Molecular Graph Encoders with LeJEPA

Self-supervised pretraining has transformed language and vision, but its value for molecular graph neural networks remains contested. We ask whether pretraining on a large unlabelled corpus improves molecular property prediction. We adapt LeJEPA, a predictor-free joint-embedding predictive architecture regularised by Sketched Isotropic Gaussian Regularisation (SIGReg), to molecular graphs, evaluating GPS and Chemprop-style D-MPNN encoders on the Wong et al. [1] antibiotic-activity dataset and ogbg-molhiv using a multi-seed, bootstrap-based protocol. Pretraining improves learned representations but does not robustly improve finetuning. A frozen probe on pretrained embeddings exceeds random initialisation on both tasks (ogbg-molhiv ROC-AUC 0.788 vs 0.665; +0.123), reaching the published self-supervised band, but this does not translate into finetuning gains. On the antibiotic scaffold split, a canonical partition is significant (delta AUPRC +0.041, p = 0.010), but the effect vanishes across five partitions (pooled +0.013, p = 0.095). Finetuning is null on the random split, ogbg-molhiv, and D-MPNN. The representational edge is nevertheless recoverable. Embeddings saturate at ~16-32 effective dimensions, whereas Morgan fingerprints improve to 1024 bits. At matched dimensionality, fingerprints lead validation (0.799 vs 0.782 at 128 dimensions) but trail shifted test scaffolds (0.759 vs 0.788). Truncating embeddings and combining them with a 1024-bit Morgan fingerprint raises ogbg-molhiv ROC-AUC from 0.805 to 0.832 (delta +0.027; 95% CI [+0.003, +0.054]; p = 0.014); an untrained encoder gains nothing (delta -0.003). Thus, pretraining supplies complementary information best realised through feature-level combination, while finetuning gains are weak and partition-dependent.

Michał Kulczykowski, Rafał Łabędzki · 0 citations

Preprint Jul 2026

Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning

Evidence is provided that cross-scale heterogeneous fusion can succeed without explicit semantic alignment when the donor contribution is sufficiently concentrated and carefully selected, and that activation-guided extraction improves the quality of the transferable donor slice while preserving the small-ratio fusion regime.

Jiahe Fan, Si Chen, Yinghao Hou et al. · 0 citations

#graph neural networks Preprint Aug 2026

Rethinking Message Passing as Retrieval for Text-Attributed Graph Learning

This work proposes RTA, a simple MLP-based framework that replaces structural message passing with label-aware retrieval and propagation and provides theoretical insights that connect retrieval-based aggregation to softmax-attention message passing and establish the robustness of retrieved-context supervision to mis-retrieved outliers.

Jintang Li, Yuhong Chen, Ruo-Fan Wu et al. · 0 citations

Preprint Aug 2026

NodeJEPA: Structure-Conditioned Latent Prediction for Node-Level Graph Self-Supervised Learning

This study offers a practical recipe for node-level JEPA-style latent prediction on graphs, and clarifies when structural conditioning helps representation learning.

Ting-He Zhang, Jian Xu, Jia-Heng Chen et al. · 0 citations