Most parameters of a Mixture-of-Experts (MoE) model belong to its experts, and standard training keeps every expert gradient in GPU memory until the optimizer step. This memory grows with the number and size of the experts, so it rises as MoE models scale. We introduce Sonar-MoE, which updates each expert tensor during...
Avni Mittal, Avinash Anand, Ashutosh Kumar et al.· 0 citations
Artificial intelligence has advanced through scale, data, and computation, yet remains bounded by its representational framework. We ask what makes intelligence open-ended. Every inferential system selects meaningful distinctions, limiting what it can learn. Within a fixed purpose, performance reaches a ceiling set by...
Simon Hegelich, Bardh Prenkaj, Habiba Sarhan et al.· 0 citations
Language models (LMs) often hallucinate by committing to confident answers rather than abstaining, even when they do not have enough information to answer reliably. A large body of existing work mitigates hallucination through detection or abstention mechanisms, but leaves open how models internally arrive at the decis...
Vy Nguyen, Zi-Qi Xu, Jeffrey Chan et al.· 0 citations
The work positions the work as a controlled reading of which parts of NCA-style computation carry the weight in language modelling, and examines a hierarchical variant of the Neural Cellular Automaton that cascades three stages with windows.
Avni Mittal, Avinash Anand, Ashutosh Kumar et al.· 0 citations
Twin Worlds (TW), a framework for improving reliability in knowledge-intensive reasoning through equivariance-based abstention, is proposed, which identifies when answers are not reliably grounded in the provided evidence and outperforms uncertainty- and sufficiency-based baselines.
Vy Nguyen, Zi-Qi Xu, Jeffrey Chan et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.