Skip to content

Can LLMs Use Relational Transformer Embeddings?

Aug 2026 · 0 citations · 10 references
Computer Science

TL;DR

It is argued that soft-token fusion requires stronger alignment objectives and schema-aware design before it can serve as a reliable route to relational prediction.

Abstract

Injecting frozen relational-encoder embeddings as soft tokens into a large language model (LLM) is a conceptually appealing fusion strategy: the encoder handles multi-table structure, the LLM handles language and reasoning, and no lossy text serialization is required. We test this hypothesis concretely by injecting embeddings from a frozen Relational Transformer (RT) into Qwen3.5-4B via a learned MLP projection and LoRA adaptation, trained first with supervised fine-tuning (SFT) on chain-of-thought reasoning traces and then with group-based reinforcement learning (GSPO). We evaluate across 10 binary classification tasks on 6 relational databases from RelBench, under four supervision regimes: single-task (ST), within-dataset (WD), cross-dataset (CD), and all-task (ALL). The hybrid model does not consistently outperform standalone RT: it is frequently below random, highly sensitive to serialization format and relational-token budget, and unstable under RL training. We report these negative results and analyze the failure modes, arguing that soft-token fusion requires stronger alignment objectives and schema-aware design before it can serve as a reliable route to relational prediction.

View source

Similar papers

#machine learning Preprint Sep 2026

Relational Attention for Data-Efficient Language Modeling

Relational BabyLM is a system submission to the BabyLM 2026 challenge that combines two cognitively motivated inductive biases in a single decoder-only Transformer, and introduces a novel symbol-retrieval mechanism that matches learned symbol libraries while adding no parameters.

Adrian Brasoveanu, Ece Takmaz, Jakub Dotlacil · 0 citations
#artificial intelligence Preprint Sep 2026

ER-JEPA: Experience Replay Improves Joint-Embedding Predictive Learning in Language Models

Large language models (LLMs) excel at token-level generation but may learn undesirable abstract semantics and lack comprehensive perception. LLM-JEPA mitigates this by aligning different views of the same underlying knowledge via a joint-embedding predictive architecture (JEPA). However, strong alignment does not neces...

Jing-Nan Pu, Zi-En Fan, Feng Lian · 0 citations
#artificial intelligence Preprint Sep 2026

Rules Amortize, Pairings Don't: Linguistic Structure Determines What Latent Task Representations Can Replace In-Context Learning

In-context learning (ICL) can be amortized into latent objects (task vectors, function vectors, context vectors) that recover few-shot behavior at zero-shot inference cost, but recent theory shows a static vector acts as a single synthetic demonstration and must fail on high-rank mappings such as word-level bijections....

Gunmay Jhingran · 0 citations
#artificial intelligence Review Aug 2026

Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability

This survey provides a structured entry point to tensorized language models and clarifies when parameter savings can plausibly translate into memory efficiency, computational efficiency, or interpretability, and introduces a metric for the compression-realization gap between theoretical memory reduction and measured sy...

M. Tarasov, Salman Ahmadi-Asl, A. D. de Almeida et al. · 2 citations

Routing Without Embeddings: Fast And Interpretable Routing With Regular Expressions

RegEXROUTE is introduced, a pipeline that uses sparse autoencoders (SAEs) to discover interpretable regular-expression (regex) features and establishes explicit, interpretable text features as a practical basis for designing and understanding LLM routers.

Yi-Fan Lu, Qi-Yue Zhang, Haotian Shan et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.