SQL-Native Vector Search at Billion Scale in Presto
TL;DR
Presto Vector Search is presented, a SQL-native distributed vector search system built on the observation that partition-based vector search decomposes naturally into relational algebra, which enables pre-filter indexing, arbitrary joins and aggregations on results, and optimizer-driven zero-shuffle distributed execution as direct consequences of the relational mapping.
Abstract
We present Presto Vector Search, a SQL-native distributed vector search system built on the observation that partition-based vector search decomposes naturally into relational algebra—partitioning as scalar functions, index construction as GROUP BY aggregation, search as equi-joins—with the local index type (FLAT, IVF, Ra-BitQ, HNSW) as a pluggable parameter within the aggregate. This decomposition enables pre-filter indexing, arbitrary joins and aggregations on results, and optimizer-driven zero-shuffle distributed execution as direct consequences of the relational mapping. A two-level API—a declarative table function rewritten by the optimizer into distributed SQL primitives—ensures both novice and expert users share the same optimized execution path. We validate across four workloads spanning three modalities (512- to 5, 120-dim) and up to 2B+ vectors, achieving 95–99% recall on workloads with ground-truth labels, with 95–99% CPU reduction over brute-force baselines and ~7,000× network reduction via codec-driven co-located execution. On the public DEEP1B benchmark (1B vectors), end-to-end BUILD+SEARCH completes in under 2.5 minutes on 200 workers with 96.6% Recall@1 against official ground truth.