Chunking Is Not the Bottleneck: Why RAG Evaluation Must Look beyond Segmentation Strategy
TL;DR
It is argued that, for QA-style retrieval on general-purpose natural-language corpora, chunking strategy is a low-variance factor, and that RAG benchmarks should report variance decomposition alongside absolute metrics so practitioners can allocate engineering effort to the factors that actually drive performance.
Abstract
: The RAG community devotes disproportionate attention to optimizing text chunking strategy. Through a factorial experiment crossing four chunking strategies, two embedding models, two chunk sizes, and two QA datasets (Natural Questions and MS-MARCO), we find that, in QA-style retrieval on two established datasets, choice of dataset (24–78% of variance) and embedding model (11–50%) together explain 67–89% of retrieval quality variance, while chunking strategy accounts for only 1–2% and is not statistically significant ( p > 0 . 21). An end-to-end LLM-as-judge evaluation confirms that chunking differences do not propagate to generation quality in this QA setting (Kruskal-Wallis p > 0 . 88). We argue that, for QA-style retrieval on general-purpose natural-language corpora, chunking strategy is a low-variance factor, and that RAG benchmarks should report variance decomposition alongside absolute metrics so practitioners can allocate engineering effort to the factors that actually drive performance.