We have 3 of 27 papers
We haven’t gathered this author’s papers yet. Follow them and we’ll fetch their work.
Not the right person? Other researchers publish under this name.
Recent successes of matrix- and curvature-based optimizers have renewed interest in how update geometry shapes learning. These methods normalize or precondition updates, changing how different components progress during training. In deep matrix factorization, the geometry that slows gradient descent (GD) favors low-ran...
This paper proposes conditioned initialization, a principled scheme that initializes attention weights to improve the spectral properties of the attention layer and shows that conditioned initialization can potentially reduce the condition number of the attention Jacobian, leading to more stable optimization.
The results show that standard transformers are rarely a local optimum in the space of architectures, and suggests that there may be room for improved architectures that better support multiple capabilities simultaneously, such as fluency and robust reasoning.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
