Skip to content

GEOSTEER: Geodesic Optimization for Activation Steering in Large Language Models

Sep 2026 · 0 citations · 29 references
Computer Science

TL;DR

GeoSteer is proposed, an optimization-based method for norm-preserving activation steering that consistently improves over state-of-the-art activation steering baselines and replaces predefined one-step edits with adaptive, geometry-aware optimization.

Abstract

Activation steering provides a lightweight way to control large language models (LLMs) by modifying their hidden activations at inference time. Among these approaches, norm-preserving steering aims to change model behavior without altering the activation norm, reducing the risk of representation collapse and degradation. However, existing norm-preserving methods are limited by predefined steering trajectories and by their reliance on one-step updates, which may fail to capture the complex structure of activation distributions. We propose GeoSteer, an optimization-based method for norm-preserving activation steering. GeoSteer formulates steering as a Riemannian optimization problem and updates activations through a sequence of small geodesic steps on the representation manifold. To avoid fixed steering directions, GeoSteer learns a nonlinear activation-space objective that distinguishes desired from undesired activations, and uses this function to adaptively guide each steering step. This multistep formulation yields smoother, more stable, and more consistent steering behavior while preserving the activation norm. Across TruthfulQA, RealToxicityPrompts, and UltraFeedback benchmarks, GeoSteer consistently improves over state-of-the-art activation steering baselines. These results suggest that norm-preserving steering can be made more effective by replacing predefined one-step edits with adaptive, geometry-aware optimization.

View source

Similar papers

#machine learning Preprint Sep 2026

Disentangling Steering Vectors

Steering Vector Dissection is proposed, a framework to explicitly isolate individual and semantically consistent features from these composite directions that tend to entangle multiple semantic and stylistic concepts into a single composite direction, leading to unpredictable steering effects.

Takeru Hiramatsu, Kyohei Atarashi, Koh Takeuchi et al. · 2 citations
Book Open access Aug 2026

A Geometric Information Bottleneck for Activation Steering

This work proposes a disentanglement-based intervention framework, termed IB-ACT, that identifies both where and how to intervene by exploiting the layer-wise geometry of representation spaces to isolate behavior-relevant information without disturbing other dimensions, and introduces a layer-selection mechanism determ...

Toan Doan, Thin Nguyen, Sunil Gupta · 0 citations
#machine learning Preprint Sep 2026

LocUS: Head Selection and Subspace Projection for Targeted Activation Steering

LocUS (Localized Unembedding Steering), a method which grounds activation steering to the model's own output vocabulary subspace within the unembedding matrix, enforces a geometric constraint that restricts the steering transformation to a specific subspace and at the same time localizes its application to a sparse sub...

Irene Tallini, Lorenzo Basile, Valentino Maiorca et al. · 0 citations
#artificial intelligence Conference Open access Sep 2026

Steering Geometry: Validating Human Value Geometry in LLM Steering Space

It is found that distribution-driven methods recover human value topologies aligned with theoretical predictions, and better geometric alignment also leads to more human-consistent transfer across values: steering one value correctly lifts compatible values and suppresses opposing ones.

Mohammad Mahdi Abootorabi, Armin Saghafian, Ali Bazshoushtari et al. · 2 citations
#machine learning Preprint Sep 2026

RW-Flow: One-Step Generation on Compact Manifolds via Wasserstein Gradient Flows

Manifold-valued data, and consequently the distributions they induce, are prevalent across many domains, ranging from the locations of geospatial events, such as earthquakes, to biomolecular torsion angles that encode information about three-dimensional structure. While diffusion and flow-based generative models have b...

Ualibyek Nurgulan, Seungwoo Yoo, Prin Phunyaphibarn et al. · 0 citations
#artificial intelligence Preprint Sep 2026

GeoSPRINT: Geometric Redundancy-Aware Step Pruning for Inference in Diffusion Trajectories

GeometricSPRINT (Geometric Step Pruning for Inference in Trajectories), a training-free framework for constructing non-uniform sampling schedules from the geometry of denoising trajectories, consistently improves over uniform DDIM (Denoising Diffusion Implicit Models) schedules at matched NFE budgets.

Arpita Joshi · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

MIT News · Artificial Intelligence Oct 23, 2026

Q&A: How AI came to model the “messy” human mind

Assistant Professor Benjamin Lindquist studies how engineers deliberately injected randomness into early AI, giving rise to deep-learning models that sometimes hallucinate.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.