GPipeANN: Taming Throughput-Latency Tradeoff in GPU-SSD ANNS
Abstract
Billion-scale Approximate Nearest Neighbor Search (ANNS) increasingly adopts a GPU-SSD architecture, where GPU fetches and processes fine-grained, SSD-resident data on-demand. However, since GPUs lack native support for I/O scheduling, existing ANNS systems are trapped in a severe throughput-latency tradeoff. They either delegate scheduling to best-effort host drivers, achieving high throughput but suffering severe latency inflation due to interference, or poll I/O completion directly inside the kernel, which achieves low latency but starves GPU utilization. This paper demonstrates that this tradeoff is not fundamental. Our key insight is that ANNS exhibits strict per-hop performance predictability in both computation and I/O, which can make scheduling lightweight enough to be onloaded directly into the GPU kernel. We thus present GPipeANN, which deterministically and lightly switches among multiple query contexts at hop boundaries to achieve perfect compute-I/O overlap. Evaluations on billion-scale datasets show GPipeANN shifts the Pareto frontier, achieving high throughput and low latency simultaneously. GPipeANN is available at https://github.com/ruc-system/GPipeANN.