Paragraph Boundaries Are Not White Space:Compression Depth as the Signature of Hierarchical Structure

Abstract

Standard positional encodings treat position as a one-dimensional reading-order coordinate, but reading order alone does not determine hierarchical textual structure. We use a hierarchical rotary positional encoding (hRoPE) that represents paragraph, sentence, and token indices as separate channels, hold the token sequence fixed, intervene on the paragraph coordinate $p_1$, and measure cross-paragraph attention with a token-distance-exact estimator. Attention is compressed in every corpus, but compression alone is not diagnostic of true structure: an architecturally identical channel with density-matched random labels is compressed too. What distinguishes real structure is the depth of compression: the real-versus-random depth gap is resolvable in two of three corpora and not in the third, and real-structure depth varies more strongly across corpora than the control's. Comparing eight corpus-only quantities across three constructs (lexical persistence, paragraph length, embedding-based coherence), none fully reproduces the cross-corpus ordering of depth, though embedding-based coherence comes closest. At the paragraph level that relation transfers as a common slope, but with the opposite sign to the corpus-level ranking: controlling for length, diversity, and position, paragraphs with more similar neighbors compress less deeply, with no detectable slope difference between any pair of corpora, while length, lexical-diversity, and position effects remain corpus-specific. Compression depth, not its location, is the reproducible signature of genuine paragraph structure in our setting.