Nimun Bajwa

Probing and Steering for Safety-Relevant Behaviours in Large Language Models

Activation probes are computed directions in a language model's residual stream, used both to detect safety-relevant behaviours and, by adding their direction back into the stream, to steer them, which power an interactive interface through which a user can score and steer text directly.