Hierarchical Memory Mamba integrates a lightweight working memory that extracts slow paragraph-level semantics from the fast sensory memory embedded in the backbone's hidden states and endows HMM cross-task generalization through parametric learning, which is not observed in other long-context enhanced Mamba variants.
Abstract
Recurrent linear attention models (RLAs) such as Mamba offer efficient linear-time sequence modeling as an alternative to Transformers, yet their fixed-capacity recurrent states limit long-sequence modeling. Drawing inspiration from hierarchical human memory, we propose Hierarchical Memory Mamba (HMM) to address this limitation. Building upon a pre-trained Mamba backbone, HMM integrates a lightweight working memory that extracts slow paragraph-level semantics (PLS) from the fast sensory memory embedded in the backbone's hidden states. The PLS is subsequently compressed into persistent long-term memory for task-relevant retrieval. The hierarchical processing of semantic information overcomes the representation bottleneck of RLAs and endows HMM cross-task generalization through parametric learning, which is not observed in other long-context enhanced Mamba variants. Evaluations on Passkey Retrieval and LongBench-E tasks demonstrate that HMM improves retrieval success by 34.3--37.1% and reasoning accuracy by 1.6--14.2% over strong Mamba-based models, while adding only 2% extra parameters and with minimal training overhead.
DART (Decoded Attention over Recurrent sTates), which retains the chunk state contributions produced by the Mamba-2 chunked scan as chunk state memories, decodes token-conditioned keys and values from these memories, and performs state-memory attention (SMA) over the resulting KV pairs.
Yixiao Qian, Song Chen, Pengkai Wang et al.· 0 citations
Deep learning for time series has progressed through successive architectural paradigms, from recurrent networks and transformers to structured state-space models, retrieval-augmented predictors, foundation models, and tool-using agents. These developments are typically studied in isolation, organized by architecture or modeling era. We argue that they can instead be viewed through a common question of \emph{how does a time-series model retain and access information beyond its immediate input?} This question is motivated by a fundamental limitation of conventional time-series modeling: information relevant to a prediction may lie far beyond a feasible input window, while compressing history into a fixed-size state can discard information that may become useful later. We formulate this challenge as a \emph{memory} problem and organize existing time-series methods along a spectrum from internal memory, encoded in parameters and fixed-size states, to external memory that is addressable, retrievable, and increasingly maintained by agents. We then develop a unified taxonomy of memory mechanisms and review three classes of external memory, including explicit modules, retrieval augmentation, and agentic stores, under a common framework for what is retained, how it is written and accessed, and how it persists. A cross-cutting analysis maps these mechanisms to time-series tasks and identifies gaps in both methods and evaluation. We conclude by outlining open problems in building memory systems that can selectively retain, retrieve, revise, and forget information as temporal environments evolve. The result is a framework for studying memory as a first-class dimension of time series modeling, independent of the underlying backbone.
M. Nguyen, H. Nguyen, Manh Nguyen et al.· 0 citations
TransMem is proposed, a lightweight inference-time parametric memory module that transforms sparse historical hidden states from a frozen LLM backbone into reusable memory representations and introduces evidence-conditioned self-distillation to learn transferable memory utilization rather than task-specific knowledge.
Knowledge Tracing (KT) is the task of modeling students' knowledge states from past learning interaction data to predict future responses. Attention-based KT models such as SAKT and AKT have demonstrated excellent performance by effectively capturing inter-problem relationships through Attention mechanisms. However, these attention-based models face efficiency limitations due to quadratically increasing computational costs as interaction sequences lengthen. Moreover, existing models struggle to effectively capture overall knowledge evolution trends across long-term learning trajectories. To address these issues, this paper proposes Hierarchical Recurrent Knowledge Tracing (HRKT), a novel knowledge tracing framework that introduces a new hierarchical recurrent design for Transformer-based KT models. By separately modeling local interaction patterns and global knowledge evolution, HRKT enables efficient long-sequence modeling and improves prediction performance. Extensive experiments on real-world educational datasets confirm that Transformer-based models augmented with HRKT significantly reduce computational cost and memory usage compared to conventional Transformer-based KT models, while achieving improved performance on long sequences.
Ju-Yeong Park, Tae-Gwon Lee, Ji-Hoon Bae· Proceedings of the 32nd ACM...· 0 citations
Recent advancements in Large Language Model (LLM) agents have largely focused on extending context windows or implementing massive Retrieval-Augmented Generation (RAG) systems to retain long-term history. However, this store-everything approach causes high computational costs and digital hoarding, paradoxically leading to digital amnesia where key emotional contexts are buried under trivial data. To challenge this paradigm, we introduce the Affective Memory Architecture, drawing from the amygdala's role in memory modulation to equip AI agents with the essential capacity to actively forget. Unlike static summarization, our framework structures multimodal inputs into an Affective Scene Graph (ASG) and dynamically adjusts the memory resolution based on emotional salience. High-arousal core memories are preserved in rich, high-resolution episodic detail; low-salience routines are aggressively downsampled using novel Optical Context Compression to minimal vision tokens; and frequently reactivated patterns are consolidated into crystallized semantic insights. Through quantitative proof-of-concept modeling, we demonstrate that systematically managing the trivial not only resolves the digital hoarding problem but actively reduces proactive interference, enhancing overall recall clarity. Ultimately, this work offers a scalable, privacy-friendly blueprint for resource-efficient AI capable of evolving with users over time, fundamentally shifting the goal of AI memory from total recall to meaningful retention.
SoYeop Yoo, Sunghoon Im· Proceedings of the 32nd ACM...· 0 citations
This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models, together with numerically stable positive-decay renormalization, to remain competitive in language modeling and improve length extrapolation on variable-digit addition.
Yi-Fan Zhang, Steve Ta, Jasper Zhang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
