Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization. Large neural networks may learn functions far simpler than their parameter counts suggest, but it is challenging to construct codes that realize this simplicity. Parameter-based methods such as quantization produce code lengths that scale with model size, insensitive to how much information the parameters store. Prequential coding bypasses this issue by compressing the training trajectory, but codes the exact data sequence regardless of how much the model learns, yielding large codes when the data has high entropy. We introduce requential coding, where a teacher model selects training samples drawn from the student's own distribution. The student's code records only these selections, which cost bits only where teacher and student disagree. The resulting code length is independent of parameter count and data entropy, and often orders of magnitude shorter than the prequential counterpart, with an advantage that grows with scale. This compression sheds light on phenomena inaccessible to prior compressors. Holding loss fixed, larger models and ensembles compress to much smaller sizes despite more parameters. Plugged into a PAC-Bayes bound, the requential code yields state-of-the-art generalization guarantees for billion-parameter LLMs, outperforming bounds built on aggressive post-training quantization even granted zero error. The bound tightens with scale in the compute-optimal regime, as models become increasingly compressible relative to dataset size. The same code predicts that models gradually overfit when trained for multiple epochs. It also isolates the learnable information in a dataset from its unpredictable, random content, revealing that lower-entropy text holds far more learnable structure than higher-entropy image data.
Distinguishing data-generating processes from finite observations is a foundational problem in nonlinear time series analysis. Compression ratios offer a model-free diagnostic of sequential complexity, yet are typically computed at a single sample size, discarding the information contained in how the ratio converges. This paper treats the convergence curve itself as the primary observable: fitting compression ratios across prefix lengths to a three-parameter scaling law, R(N) = a + bN-c, yields three diagnostics from a single analysis. The asymptote a provides a rank-faithful entropy rate proxy (Spearman ρ = 0.977 across 36 processes with known analytical rates), performing on par with the best point-estimate measures. The convergence exponent c captures how quickly the compressor exhausts learnable structure-a dimension not captured by any of the seven benchmark complexity measures tested here, including effort-to-compress (maximum |r| = 0.34). Among generators sharing identical entropy rates, c still varies substantially (mean range 0.173 across 22 same-entropy clusters), distinguishing processes that every point-estimate method considers equivalent. The compression consistency (R2 of the scaling-law fit) serves as a compositional heterogeneity diagnostic: R2 degrades systematically when data from multiple generating processes are temporally concatenated, but not for independent and identically distributed mixtures of the same generators. Deterministic chaotic maps are flagged by Bayesian-information-criterion preference for the logarithmic convergence form motivated by Lempel-Ziv (LZ) asymptotics, manifesting as distinctively low c (<0.40) for generators in the chaotic regime under the power-law parameterization. The framework is validated on 152 synthetic generators spanning 25 families and 105 real-world datasets. Code and data are publicly available at https://github.com/generative-structure/cbad-finite-size-scaling.
TESLA, an activation defined as a learnable combination of sine and cosine terms, enabling explicit control over polynomial degree and selective amplification of high-order components is proposed, indicating that activation-level degree control transfers to more general vision workloads.
Daehwa Ko, Jae-Hwan Kim, Seunghyun Ham et al.· 0 citations
This work identifies and formalizes the principle that organizes the Great Inversion, the Great Inversion: allocation-flexible coding rewards energy concentration, whereas the grouped shared-scale quantization a deployed matrix instruction performs rewards within-group flattening.
Post-training quantization compresses large language models (LLMs) by storing their weights at reduced precision, and each quantized weight introduces an error into the hidden states. Naively, these errors should accumulate with depth and corrupt next-token prediction; randomly initialized models accumulate these discrepancies rapidly, whereas quantized pretrained models accumulate much less hidden-state error and largely maintain downstream task performance, even though they were never trained with quantization noise. This raises the question we address: why does post-training quantization work? Comparing full-precision and quantized forward passes, we identify two mechanisms that characterize pretrained quantization robustness. First, the error a layer newly introduces tends to oppose the error it inherits from the layer's input. The two cancel partially such that the discrepancy between full-precision and quantized passes grows slowly. This counteracting residual interaction develops during pretraining. Our quantitative analysis identifies it as a major factor slowing hidden-error growth. Second, LM-head geometry preferentially preserves the scores and probabilities of high-ranked tokens, which typically represent the model's most confident predictions. Together, these mechanisms explain why quantization error that passes through numerous layers can still produce only small output changes, and we verify the findings across models and quantization settings.
Yu-Xiang Chen, Michael Beyer, Jun Zhu et al.· 0 citations
By introducing RSLM (Rotated Scaled Lloyd-Max), a family of training-free vector quantization codecs compressing embeddings to 1--4 bits per dimension, this work reduces memory cost and memory bandwidth of a typical large-scale Approximate Nearest Neighbor (ANN) search system, while reducing its complexity and keeping or improving recall across multiple benchmark datasets.
Rastislav Lenhardt, Teodora Dobos, Thomas Vecchiato et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
