Learning diverse attacks on large language models for robust red-teaming and safety tuning
This work proposes to use GFlowNet fine-tuning followed by a secondary smoothing phase, to train the attacker model to generate diverse and effective attack prompts, and finds that the attacks generated by the method are effective against a wide range of target LLMs, both with and without safety tuning, and transfer we...