Skip to content
Book Open access

GPipeANN: Taming Throughput-Latency Tradeoff in GPU-SSD ANNS

Nov 2026 · Proceedings of the 2026 ACM SIGOPS Annual Technical Conference · 0 citations · 34 references

Abstract

Billion-scale Approximate Nearest Neighbor Search (ANNS) increasingly adopts a GPU-SSD architecture, where GPU fetches and processes fine-grained, SSD-resident data on-demand. However, since GPUs lack native support for I/O scheduling, existing ANNS systems are trapped in a severe throughput-latency tradeoff. They either delegate scheduling to best-effort host drivers, achieving high throughput but suffering severe latency inflation due to interference, or poll I/O completion directly inside the kernel, which achieves low latency but starves GPU utilization. This paper demonstrates that this tradeoff is not fundamental. Our key insight is that ANNS exhibits strict per-hop performance predictability in both computation and I/O, which can make scheduling lightweight enough to be onloaded directly into the GPU kernel. We thus present GPipeANN, which deterministically and lightly switches among multiple query contexts at hop boundaries to achieve perfect compute-I/O overlap. Evaluations on billion-scale datasets show GPipeANN shifts the Pareto frontier, achieving high throughput and low latency simultaneously. GPipeANN is available at https://github.com/ruc-system/GPipeANN.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.