To Tier, or Not? Evaluating Tiered Memory Systems on the Basis of Cost

Abstract

CXL memory expansion is often motivated by lower-cost capacity, but tiering systems are usually evaluated by their performance, not by their cost savings. This disconnect leaves a critical question unanswered: do the cost savings of slow CXL memory make up for the loss in performance? To address this, we argue for evaluating the cost-effectiveness of tiered memory under different fast-to-slow memory ratios. We develop a model to compute the break-even CXL cost, which is the maximum CXL cost at which a tiered configuration is just as cost-effective as an all-local-DRAM baseline for a given workload. This approach helps quantify the cost of hardware while avoiding dependence on unknown CXL device pricing. We apply our model to memory-intensive workloads on a real CXL server using AutoNUMA memory tiering, based on TPP. Across our workloads, break-even costs range from above local-DRAM cost in favorable configurations to below zero in unfavorable ones. The former indicates that CXL can be cost-effective even without a discount relative to local DRAM; the latter indicates that even free CXL memory would not offset the slowdown. We find that the economic case for using CXL is strongest when it helps avoid scaling local memory with expensive dense memory, and that CXL-heavy memory configurations often used for evaluation are not the most cost efficient and require very cheap memory.