The results suggest that the combination of sparse representations, local learning, and persistent memory is a promising direction for continual learning, while motivating further investigation into the respective roles of learning rules, representations, and architectural design in mitigating catastrophic forgetting.
Abstract
We introduce CMP (Cognitive Memory Primitive), a continual-learning architecture that repre?sents inputs as sparse relational codes, stores them in a two-tier competitive memory, and learns through local updates without end-to-end backpropagation through its feature-generating system. We investigate whether combining sparse representations, local learning, and persistent memory can reduce catastrophic forgetting relative to conventional backpropagation-based continual?learning approaches. On a controlled domain-incremental byte-level language modeling protocol, CMP demonstrates substantially lower backward transfer than a parameter-matched Trans?former trained with online Elastic Weight Consolidation (EWC). Across a three-seed replicated 15-domain experiment, CMP exhibits stable forgetting behavior, while separate head-to-head comparisons and domain-order analyses show consistently lower forgetting than the evaluated Transformer baseline under the reported experimental settings. We report these findings alongside a substantial single-domain accuracy gap relative to the Transformer, a null result on a vision benchmark, and a documented failure to combine CMP with an independent accuracy-improving mechanism, reflecting our commitment to reporting both positive and negative outcomes. These results suggest that the combination of sparse representations, local learning, and persistent memory is a promising direction for continual learning, while motivating further investigation into the respective roles of learning rules, representations, and architectural design in mitigating catastrophic forgetting.
We introduce CMP (Cognitive Memory Primitive), an architecture that represents inputs as sparse relational codes, stores them in a two-tier competitive memory, and learns entirely through local, gradient-free updates, with no backpropagation anywhere in the network. We use this architecture to test a specific hypothesis: that catastrophic forgetting, usually treated as a training-time defect to be patched with replay or regularization, is instead a structural consequence of how backpropagation assigns credit and that a learning rule that is local and sparse by construction should resist it without a patch. On a controlled domain-incremental protocol across 15 text domains, three-seed replicated, CMP's backward transfer is 15-19x better than a matched-size Transformer trained with online EWC, and the result survives a domain-order control (reported as a range, +0.24 to +0.44, rather than a single figure). We report this alongside a real, substantial accuracy gap versus the Transformer baseline, a null result on a recognized vision benchmark, and a diagnosed, unresolved failure attempting to combine this architecture with a separate mechanism that improves raw accuracy, disclosed because an honest negative result is more useful than an omitted one. The central claim is narrow and falsifiable: local, sparse, non-backpropagation learning measurably resists catastrophic forgetting better than backpropagation with its standard fix, under conditions we state precisely.
Current mainstream deep learning techniques exhibit an over-reliance on extensive training data and a lack of adaptability to the dynamic world, marking a considerable disparity from human intelligence. To bridge this gap, Few-Shot Class-Incremental Learning(FSCIL) has emerged, focusing on continuous learning of new categories with limited samples without forgetting old knowledge. Existing FSCIL studies typically use a single model to learn knowledge across all sessions, inevitably leading to the stability–plasticity dilemma. Unlike machines that usually consolidate all categories into a single parameter space, cortical memory organization suggests that different types of knowledge can be distributed and organized across specialized cortical regions. Inspired by this organization principle, our paper aims to develop a method that learns independent models for each session. It can inherently prevent catastrophic forgetting. During the testing stage, our method integrates Uncertainty Quantification (UQ) for model deployment. Our method provides a fresh viewpoint for FSCIL and demonstrates the state-of-the-art performance on CIFAR-100 and mini-ImageNet datasets.
Renye Zhang, Yi Yin, Jinghua Zhang· Entropy· 0 citations
Humans retain memories of individual experiences for a lifetime, an ability attributed to a complementary learning system in which a fast process encodes episodes and a slow process integrates them into semantic knowledge. In classical Hebbian models such as Hopfield networks, memory traces are superposed in shared weights. This makes learning naturally continual but causes strong interference among correlated memories, a failure that reappears as catastrophic forgetting in deep networks. Here we use a large language model as a model system for continual episodic memory, with its pretrained weights supplying the semantic context in which new episodes are embedded. Fast learning is implemented by a hippocampus-like module that assigns each episode to a dedicated, extremely sparse low-rank adapter; competitive gating then selects among these separated traces during recall. Across streams of up to 1,000 factual and autobiographical episodes, each adapter requires only 2–3 parameters per token while preserving excellent recall. An internal retrieval-augmented generation mechanism reconstructs the selected episode in context and supports high-accuracy question answering over stored memories. Finally, slow cortical consolidation is modeled by fine-tuning the base weights through batch replay, enabling reconstruction and direct question answering without episodic adapters. Together, fast storage and slow consolidation implement both components of a complementary learning system within a single language model, yielding a neural-network model that stores, recalls, and consolidates naturalistic episodic memories, thereby capturing key functional features of human memory.
Xu Pan, Ely Hahami, R. Siegelmann et al.· bioRxiv· 0 citations
This review provides a structured overview of major CL paradigms, including task-incremental, domain-incremental, class-incremental, online, multimodal, and federated CL, and examines the theoretical foundations of CL, particularly the stability–plasticity dilemma, catastrophic forgetting, transfer dynamics, and representation learning.
Continual learning (CL) aims to incrementally update machine learning models from a stream of data without forgetting previously acquired knowledge. CL is highly relevant in many real-world applications such as manufacturing, where storing historical data for retraining is often infeasible due to volume, governance, or system constraints. Yet, a common challenge in continual learning (CL) is catastrophic forgetting, where the performance on old tasks drops after new, additional tasks are learned. In this paper, we propose a novel framework called ReCL to slow down forgetting in CL. Our framework exploits an implicit bias of gradient-based neural networks due to which these converge to margin maximization points. Such convergence points allow us to reconstruct old data from previous tasks, which we then combine with the current training data. Our framework is flexible and can be applied on top of existing, state-of-the-art CL methods. We first demonstrate the performance gain from our framework across a large series of experiments on three established public CL datasets (MNIST, CIFAR10, TinyImageNet), a public industrial dataset (SECOM), and across two different scenarios (class incremental and domain incremental learning). Then, we evaluate the performance of ReCL for predictive maintenance in a manufacturing environment at Bosch, a global engineering company, using an internal real-world time-series feature dataset captured from a high-volume precision-machining process. Lastly, we apply our framework to streaming machine data from real-world industrial data at Bosch. Across all our experiments, we find large performance gains through ReCL. To the best of our knowledge, our framework is the first to address catastrophic forgetting by leveraging models in CL as their own memory buffers.
Pascal Janetzky, T. Schlagenhauf, Michael Klar et al.· Proceedings of the 32nd ACM...· 0 citations
As large language models (LLMs) become increasingly capable, the next question is how can we enable models to continually learn? Today, the field largely frames this as a problem of context management and mitigating forgetting. We argue this framing is incomplete: continual learning is fundamentally about increasing model competence as the world changes. We disentangle this change along two axes -- space, where the model encounters new domains, and time, where the underlying data drifts under a fixed task. This framing lets us study continual learning under realistic conditions: new domains arrive over time, facts drift past their training cutoff, and agentic interactions accumulate state across episodes. To evaluate methods under this setting, we recast widely used LLM benchmarks as sequential problems and introduce a single mechanism-agnostic protocol that compares prompt-based methods (GEPA, ACE), supervised learning (SFT, SDFT), reinforcement learning (GRPO, SDPO), and context compression (Cartridges, In-place TTT). Prompt-based methods fit each new stage quickly but degrade on future tasks. Distillation-based methods accumulate knowledge stably but struggle to update outdated facts. Context compression improves efficiency without substantially improving the ability to learn new tasks. Online reinforcement learning adapts most effectively to knowledge updates but remains sensitive to noisy reward signals. Overall, our results suggest that continual learning is not a single capability: different patterns of environmental change require fundamentally different update behaviors, determining when adaptation must be learned inside model weights and when it can be achieved through external scaffolding. We hope that understanding where each method succeeds and fails will guide the design of stronger continual learning systems.
A. Harrington, Nayan Saxena, Michael Murphy et al.· 1 citation· ⚡1