What makes a deep network train, and what shape is its loss surface? We answer both with one construction at two scales: a polynomial activation f (z) = z + {\lambda}z 3 , clamped and projected onto the unit sphere, plus an unnormalized identity path. As a deep stack it trains to one hundred layers on MNIST with no nor...
Under this condition, it is proved exponential convergence to a unique fixed point and its bias relative to the unregularized Bellman solution and it is shown that the same mechanism extends to entropy-regularized off-policy control through soft Q-learning.
Transformers can perform in-context reinforcement learning without parameter updates. However, existing theoretical constructions often require context sizes that grow with the number of sampled transitions, leading to increasing memory costs in large-scale problems. This raises a fundamental scalability question: how...
Yajie Zhao, Yingjun Mo, Zhiyong Wang et al.· 0 citations
Efficient sequence modeling remains a fundamental challenge in machine learning. Despite the success of Transformers, recurrent neural networks (RNNs) have recently re-emerged as efficient alternatives. Particularly, Echo State Networks (ESNs) offer a compelling framework of randomized RNNs that have been extensively s...
Mario Mallea, À. Nebot, Franscisco Mugica· 0 citations
Reach audiences
Advertise in front of researchers, engineers, and readers.
Federated fine-tuning enables large language models (LLMs) to learn collaboratively from distributed client data, but malicious client updates can compromise global model safety. As federated tasks evolve, changing malicious update patterns challenge the ability of existing defenses to maintain reliable risk detection...
Shuaijie Ma, Yongkang Hu, Yu Cheng et al.· 0 citations
Energy-based models can represent distributions over discrete structures such as molecular graphs, but an energy that is accurate at equilibrium need not yield accurate samples when the sampler stops after a fixed number of steps. We introduce finite-horizon energy learning, which adapts the acceptance energy of a Metr...
Amartya Banerjee, Xingyu Xu, Caroline Moosm\"uller et al.· 0 citations
Neural networks operate naturally on finite-dimensional vectors, yet many problems require reasoning about higher-order objects such as distributions, functions, and graphs. Such objects capture relationships within a ground set and rarely admit an explicit compact representation. Moreover, we typically want a compact...
Radovan Halu\v{s}ka, Christopher Solinas, Nathan R. Sturtevant et al.· 0 citations
Population dynamics are often observed only as unpaired snapshots, which leave the evolution between measurements underdetermined. Second-order regularization can constrain it, but a penalty on the accelerations of sampled particles is a prior on joint particle paths, and different particle representations of the same...
Memory-efficient subspace optimization has emerged as a practical approach to reducing the memory cost of LLM training while preserving optimization performance. Previous algorithms typically rely on the leading singular subspace of stochastic gradients, yet the gradient noise may dominate the subspace in later trainin...
Sifan Yang, Hongyi Tao, Dingzhi Yu et al.· 0 citations
Spatio-temporal forecasting is fundamental to many real-world systems, yet deployment environments are inherently varying: temporal distributions may evolve, spatial domains may change, and observation availability may shift from training conditions. Existing models are often tightly coupled to their training environme...
Shengnan Guo, Jinjun Cheng, Xiaowei Mao et al.· 0 citations
Large protein language models provide a powerful framework for learning sequence-function relationships and guiding experimental design. However, architectural improvements are underexplored, especially in low-compute regimes of relevance for resource-constrained research laboratories. We show that the training cost fo...
Alberto Martínez, Natalia Flechas Manrique, Llorenç Espinosa-Portalés et al.· 0 citations
Goal misgeneralization is a common failure mode in which an agent behaves competently out of distribution but does not pursue the intended goal. However, we still have a weak understanding of what happens to the intended goal in misgeneralizing agents. Across three Procgen environments and five robot tasks, we find thr...
Julia Barth, Armin Sommer, Kunvar Thaman et al.· 0 citations
