Prompt-based adaptation offers several ways to specialize a frozen language model, ranging from natural-language prompt optimization to learned continuous prompts. These methods are usually compared through downstream performance, although similar outputs need not arise from similar internal computations. We study how...
Mikhail Aleksandrov, Nail Bashirov, Maksim Ivanov et al.· 0 citations
Block diffusion language models keep a large key-value (KV) cache throughout generation and attend to it at every denoising step, limiting both memory capacity and generation speed. Reducing these costs requires deciding which past tokens to use for denoising the current block (selection) and which to keep in memory fo...
G. Molodtsov, Ekaterina Alimaskina, Е. И. Усков et al.· 0 citations
This work proposes LionMuon, which retains the effectiveness of Muon steps while considerably cutting the averaged iteration cost, similar to sign-based methods, and proves sharp complexity bounds under heavy-tailed noise which are governed by period-averaged smoothness and noise that interpolate between Muon's and Lio...
Arman Bolatov, A. Riabinin, Nikita Kornilov et al.· arXiv.org· 2 citations
This work introduces HARP (Hadamard-preconditioned Adaptive Rotation Processor), a learnable structured two-sided orthogonal processor that replaces fixed Hadamard mixing while preserving exact full-precision equivalence.
A. Zagitov, G. Molodtsov, A. Beznosikov· arXiv.org· 4 citations
LoRA-TSD, an optimizer that treats every LoRA step as a tangent vector of the fixed-rank matrix manifold and takes the spectral-norm steepest-descent step of Muon inside that tangent space, mapping the result back to the factors through a retraction native to the LoRA parametrization, is introduced.
Dmitrii Andriianov, Andrey Veprikov, A. Beznosikov· 0 citations
This work develops a self-contained methodology for learning parameter-efficient rotational transformations based on Riemannian optimization and empirically validate the proposed rotation-based steering scheme, demonstrating its superiority in intervention efficiency.
Kirill Bunin, Dmitry Bylinkin, Vladimir Aletov et al.· 0 citations
A novel white-box attack inspired by locate-then-edit approaches from the field of Knowledge Editing is proposed, modifying the editing framework by incorporating as- sociative knowledge retrieved from the model, thereby extending constraint removal to an entire thematic category rather than being limited to prompts fr...
R. Maksimov, Vladimir Aletov, V. Solodkin et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.