This work proposes continuous-query LMLM (CO-LMLM), where the KB pairs continuous keys with textual knowledge values, a significant departure from prior reliance on relational KB and queries, which outperforms prior LMLMs and vanilla LLMs in both perplexity and factual precision.
Abstract
Limited memory language models (LMLMs) externalize factual knowledge during pretraining to a knowledge base (KB), rather than memorizing it in their weights. During generation, the model then fetches knowledge from the KB as needed. This recently introduced paradigm provides multiple advantages, including knowledge control capabilities that remain beyond conventional LLMs. We propose continuous-query LMLM (CO-LMLM), where the KB pairs continuous keys with textual knowledge values, a significant departure from prior reliance on relational KB and queries. CO-LMLM generates flexible vector queries at minimal cost, while still integrating human-readable and attributable retrieved knowledge into its generation. We pair this design with an annotation pipeline that tags free-form factual spans in arbitrary text, removing prior work's restriction to Wikipedia. Across pretraining on Wikipedia and FineWeb-Edu and at multiple model scales, CO-LMLM outperforms prior LMLMs and vanilla LLMs in both perplexity and factual precision. At 360M scale, this includes lower perplexity than models pretrained on 40x more data, and SimpleQA-verified performance that is in line with gpt-4o-mini and higher than Claude Sonnet 4.5.
LMEnt is released to support studies of knowledge in LMs, including knowledge representations, plasticity, editing, attribution, hallucinations, and learning dynamics, finding that entity co-occurrence and mention forms—which are difficult to study with existing tools—affect learning trends.
Daniela Gottesman, Alon Gilaie-Dotan, Ido Cohen et al.· Transactions of the Associat...· 0 citations
Large language models (LLMs) are increasingly used in personalized applications, where responses must align with individual user preferences, histories, and profiles. A common approach is to inject user information into the prompt at inference time. However, existing methods typically rely on flat profile representations and similarity-based Top-K retrieval, which often introduce noisy or marginally relevant context under limited prompt budgets. In this paper, we propose PersMem, a user profile memory framework for LLM personalization that addresses three key questions: what user information to store, how to organize it, and how to use it effectively during generation. First, we construct a structured user profile memory by extracting entities and concepts from user profiles. Second, we organize the memory into a hierarchical structure that separates short-term interactions from long-term knowledge, enabling scalable and coarse-to-fine retrieval. Third, we introduce a learning-based memory gating mechanism that adaptively selects long-term memory units under prompt budget constraints and directly optimizes answer quality through evaluated signals. Extensive experiments on multiple personalization tasks demonstrate that our approach consistently improves personalization effectiveness while reducing prompt length, compared with existing prompt-based and retrieval-based baselines.
Yangxu Liao, Yongheng Deng, Tianyuan Jiang et al.· Proceedings of the 32nd ACM...· 0 citations
These results show that long-context memory can be organized along the layer axis, not only the token axis, and expose both the benefits of bounded retrieval and its in-window compression tax.
Han-Lin Liu, Xuan Qi, Chunyu Liu et al.· 0 citations
The results demonstrate that suppressing entity-linked supervision during pretraining induces a shift in epistemic behavior: KLLMs rely less on parametric knowledge and more on external evidence, leading to improved reliability under realistic conditions, suggesting that pretraining-time control over knowledge acquisition can complement retrieval-augmented and tool-based systems by providing a more evidence-sensitive base model.
Roi Cohen, Yvan Carr'e, Nick Lechtenborger et al.· 0 citations
An LLM reasoning framework with hierarchical relational retrieval for large-scale knowledge updating, named G-HiRel, which achieves superiority in terms of accuracy and interpretability and handles the knowledge inconsistency between the KG and LLM to obtain entity independence.
Yudai Pan, Jiajie Hong, Tianzhe Zhao et al.· Annual Meeting of the Associ...· 0 citations
This work proposes MARS, a scalable knowledge graph question answering (KGQA) approach that requires no model fine-tuning, and performs a structured retrieval procedure that links question entities to the KG and iteratively retrieves relevant next-hop information.
Nikit Srivastava, Daniel Vollmers, René Speck et al.· 0 citations