We have 4 of 29 papers
We haven’t gathered this author’s papers yet. Follow them and we’ll fetch their work.
Not the right person? Other researchers publish under this name.
A uniform compressibility theorem for deep multilayer perceptrons with analytic activations is proved, which involves a novel derivative-matching technique which is aware of the low-dimensional input, and a layer-wise reweighting that preserves the input-output mapping.
This work proposes a curvature-conditioned multiscale momentum method with sphere constraints, which significantly accelerates Muon across diverse architectures (dense, MoE) and model sizes (0.12B--2.3B parameters).
This work presents a systematic study of scale vectors in LLMs from the perspectives of expressivity, optimization, and architectural structure, and proposes three lightweight and complementary improvements to scale vectors: branch-specific heterogeneity, improved placement around linear mappings, and magnitude-directi...
This work proposes Mixture of Activations (MoA), a token-adaptive FFN design that mixes a dictionary of activation functions using lightweight input-dependent gates while sharing the same linear projections, suggesting that token-adaptive activation mixing is a simple and effective mechanism for improving FFN expressiv...
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
