Skip to content

CACHEFORGE: LLM-Guided End-to-End Generative Cache Replacement Policy for Performance and Hardware Efficiency

Oct 2026 · 0 citations · 26 references
Computer Science

TL;DR

CACHEFORGE is the first framework to evolve cache-replacement policies end-to-end by embedding a large language model inside a governed hardware-aware loop and enforces feasibility through reward shaping, structural checks, dynamic mutation, temperature scheduling, and cross-policy crossover.

Abstract

Modern cache replacement designs saturate because they operate within fixed representational structures, hand-crafted and heuristic based feature-engineered predictors, or offline imitation models that cannot generate new decision logic on their own. At the same time, replacement is shaped by the causal interaction of prefetching, thrashing, spatial locality, and access-type behavior, producing an enormous design space that is difficult to traverse manually. Prior approaches typically rely on heuristics, parameter tuning, or imitation of an offline optimal policy, capturing correlations rather than synthesizing new mechanisms. As a result, their performance gains often plateau and they overfit under dynamic workload conditions. CACHEFORGE is the first framework to evolve cache-replacement policies end-to-end by embedding a large language model inside a governed hardware-aware loop. In each iteration, the LLM proposes new C++ replacement logic, the policy is evaluated under a trace-based CRC-2 ChampSim simulator, and the framework enforces feasibility through reward shaping, structural checks, dynamic mutation, temperature scheduling, and cross-policy crossover. This closed-loop generation-evolution loop specifically designed for cache replacement policy enables the discovery of compact policies that satisfy hardware constraints while exploring algorithmic transformations beyond fixed predictor structures. Across SPEC CPU2006, CACHEFORGE outperforms all CRC-2 baselines. It improves the total hit rate by 27.36%, 19.69%, 13.72%, 13.15%, 11.83%, and 5.73% over MPPPB, ReD, Hawk-eye, SHiP++, LIME, and LRU, respectively. On memory-intensive workloads, it increases IPC by 10.15%, 7.89%, 6.34%, 3.64%, 3.12%, and 2.71% over LRU, MPPPB, LIME, ReD, SHiP++, and Hawkeye.

View source

Similar papers

Preprint Oct 2026

CacheFit: The Rising Cost of Cache Replacement and Constraint-Driven Redesign of Last-Level Cache Replacement Policies

The latency, power, and area costs of LLC replacement remain poorly explored not because simulators cannot be instrumented--event counters feeding a power model are routine--but because no cost model takes a replacement policy as input. CACTI and McPAT price regular structures, a cache or a core, not the small, oddly s...

Bita Aslrousta, Kaushal Mhapsekar, Darsh Asher et al. · 0 citations
Book Open access Oct 2026

SimServing: Native-Execution Simulation for Evolution-Resilient LLM Serving Configuration Tuning

Large Language Model (LLM) services expose a large configuration space that critically affects performance, yet tuning these configurations on hardware (e.g., NVIDIA GPUs) is prohibitively expensive. Simulation offers a scalable alternative, but state-of-the-art simulators suffer from a fundamental software engineering...

Yu-Jie Huang, Jia-Zhen Gu, Zhihan Jiang et al. · 0 citations
Open access Sep 2026

STATE-AWARE PREFETCHING AND CACHE POLICIES FOR LATENCY-OPTIMIZED WEB EXPERIENCES

Modern web applications demand sustained low latency under workloads that shift across users, devices, sessions, and network conditions. Classical cache replacement policies such as Least Recently Used (LRU) and Least Frequently Used (LFU) treat every cached object identically and ignore the cost-of-miss heterogeneity...

Akshatha Madapura Anantharamu · 0 citations
Book Open access Sep 2026

PINCH: Predictive Importance-Sampling-Informed Cache for I/O-Bound DNN Training

Pinch, a cache middleware that decouples clairvoyant and importance sampling at the granularity of a planning horizon, enabling planned cache retention and bounded prefetch on DNN training on large datasets.

Yu-Chen Liu, Shu Yin · 0 citations
Open access Oct 2026

JumpStart: Alleviating the Front-End Bottleneck in Modern x86 Processors

With the stagnation of single-threaded performance, the need for optimizations targeting sequential code performance is greater than ever before. Modern workloads have also been growing in code footprint, exposing a bottleneck in the processor front-end, specifically its inability to maintain a sufficiently high fetch...

L. Moody, B. Godala, Gilles A. Pokam et al. · 0 citations
Open access Aug 2026

Rethinking Cache Eviction: Low-Overhead and Precise Learning-based Design

Caches reduce latency and network traffic, and cache performance largely depends on the eviction policy. Eviction effectiveness is typically measured by byte and object miss ratios. Although learning-based eviction policies can reduce misses, their high computational overhead limits practical deployment. This work pres...

Qian Wang, Chang Xu, Wen-Bin Zhou et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.