Automating AI research creates a positive feedback loop between software capability and research effort. We study a stochastic realisation of this loop in which independent, positive, multiplicative improvements arrive at a rate proportional to a power of current capability. Within this model, randomness leaves the det...
Xiao-Yu Li, Zheng-Guang Gao, Chi-Wun Yang et al.· 0 citations
Parameter-efficient fine-tuning has hundreds of methods, usually compared by parameter count and benchmark score. We compare them as maps. A pretrained checkpoint is a base point in weight space, and a weight-space method is a smooth map from a small parameter space that starts at that point. These maps form a slice ca...
Xiaoyu Li, Zhizhou Sha, Chi-Wun Yang et al.· 0 citations
Hyper-connections widen the residual stream of a Transformer to $n$ parallel streams. Their manifold-constrained version (mHC) mixes the streams at each layer with a doubly stochastic matrix, which it computes by Sinkhorn normalization of exponentiated logits. We give a geometric theory of this design on the Birkhoff p...
A reward can identify a correct answer while leaving the computation needed for new inputs undetermined. We study how pretraining and midtraining supply the information and computation that make reward adaptation effective. In sequential state computation and contextual memory, we characterize mechanisms that agree on...
The minimax law is proved, showing that the two resources are governed by one predictive-energy spectrum in a positive-entropy autoregressive retrieval source and giving exponent-adaptive allocation, finite-precision realization, and compute-precision laws under two-sided arithmetic assumptions.
Results motivate CertKV, a training-free compressor that reserves one tail-summary slot per head and allocates the rest by value dispersion, which is top-two in seven of nine LongBench-v2 settings, remains in the leading compressed tier on 128K RULER, and realizes a ten-fold cache budget in a packed Llama prototype.
Chi-Wun Yang, Xiaoyu Li· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
