User Space Buffer Cache with the Bells and Whistles of a Kernel Page Cache

Abstract

Modern NVMe SSDs push the I/O bottleneck into software. Kernel-bypass frameworks accelerate direct I/O but leave buffered I/O, the dominant mode in practice, unoptimized. We show that small buffered I/O (32 B–4 KB) incurs a 2–15 × kernel overhead and dominates real server and mobile workloads. Existing user-space caches fail to simultaneously meet four key properties: file-level caching, low access overhead, hardware agnosticism, and efficient memory usage. While the first three can be achieved by moving the page cache to user space, efficient memory usage remains elusive due to the lack of system-wide coordination across isolated process address spaces. This paper presents Nexus, a quasi-user-space I/O stack with a split architecture: a user-space data plane and a lightweight kernel control plane. The data plane performs cache lookups and I/O submission with zero system calls, thereby delivering file-level caching, low access overhead, and hardware agnosticism. The control plane, in turn, enables efficient memory usage through shared data structures and in-kernel page reclamation that coordinate global allocation and reclamation across all processes. Our evaluation shows that Nexus achieves up to 3.4 × throughput for small sequential reads and 1.5–1.8 × for random reads over ext4 with dynamic and efficient memory usage.