The law under real (COVID-19) workload drift is validated, the gate in PostgreSQL/pgvector at a 0.33% ingest tax is implemented, and the bound is turned into a provisioning rule that sizes the sentinel budget per emerging region.
Abstract
In a vector database serving production-scale retrieval, a single inserted document can be retrieved for an anomalously large share of the query workload -- a retrieval hub -- and dominate the evidence returned for an entire topic. An emerging defense guards against this at ingest with an admission check: it maintains a set of sentinel queries and admits a document only if its reverse-kNN count against them stays below a threshold tau. Under workload drift this sentinel set is a query-aware auxiliary index that must be maintained online, and we study the cost that maintenance imposes on the ingest path. We identify a structural limit -- coverage is not redundancy: a monitor stops promoting sentinels once a region is covered, but the predicate rejects a hub only once tau sentinels witness it, so exposure has an observation-limited floor that no reduction in update or enforcement latency can close. On real HNSW, IVF-Flat, and IVF-PQ indexes over an 8.8M-vector MS MARCO corpus this floor is only a best case: as index recall falls, exposure and churn rise above it, and below recall ~0.5 the gate stops containing altogether -- worst on the memory-compressed IVF-PQ used at billion scale -- while a recall-aware witness probe restores containment at a fixed O(|S|d) admission cost, under 0.1% of the ANN insert. We validate the law under real (COVID-19) workload drift, implement the gate in PostgreSQL/pgvector at a 0.33% ingest tax, and turn the bound into a provisioning rule that sizes the sentinel budget per emerging region. A count test contains the hub where retrieval-time score normalizers (NNN, QB-Norm) do not, and a pre-registered causal suite isolates the missing-coverage mechanism from retrieval fragmentation across two embedding families (BGE-1024, E5-768).
The one signal that separates an attack from legitimate niche ingestion -- a query's demand -- is invisible before retrieval, which is also the escape: a retrieval-time detector that observes demand catches 100% of the attacks at the same 1% false-positive rate.
The cross-encoder study shows that thresholds do not transfer between embedding models, and LFU is the strongest simple default in this protocol; deployment decisions should first establish answer validity and then test sub-point policy differences with exact search.
Y. Kulkarni, Shubham Harkare, A. Babu· 0 citations
This approach is pre-retrieval and query-centric, making it robust in heterogeneous, case-like corpora and applicable across domains such as legal, healthcare, finance, and enterprise search.
In production, agentic systems answer questions over data that lives in several places and keeps changing: licenses are reassigned, users offboarded, prices changed, contracts renewed. Existing retrieval benchmarks freeze the data, so they cannot ask whether an agent's answer is still true, only whether it found the ri...
V. Singh, Preeti Priyam, Gautam Bhowmick· 0 citations
Approximate nearest neighbor (ANN) search is an integral part of contemporary vector databases; however, the performance of this algorithm highly depends on the specifics of data, recall requirements, storage capabilities, and query workload. All these factors complicate ANN even further in multi-vector databases, wher...
S. Salunkhe, K. Vayadande, H. Khandagale et al.· MethodsX· 0 citations
Modern web applications are increasingly reliant on Object-Relational Mapper (ORM) frameworks to bridge the gap between high-level application logic and relational databases. However, in standard Create, Read, Update, and Delete (CRUD) architectures, the use of a unified data model often leads to significant performanc...
Justin Evan Surya, Kenny Jingga· International Conferences on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.