Jack Hopkins

Overthinking: Amplifying Reasoning Weights to Extract Learned Secrets

Overthinking is introduced, the process of using reasoning task vectors to amplify the propensity to think out loud of reasoning models, and reasoning amplification may surface secrets or unintended behaviors acquired during training up to $10\times$ more frequently than the original reasoning model.