From Evaluation to Optimisation: Hierarchy-Aware Training Signals for CWE Prediction in Python
TL;DR
Testing three delivery mechanisms–supervised fine-tuning, a dual-head classification loss, and reinforcement learning with a dense reward derived from the normalised penalty finds that supervised approaches consistently regress below the zero-shot baseline under distribution shift, while GRPO succeeds and generalises across independent training seeds.
Abstract
The original ALPHA benchmark introduced a taxonomy-aware penalty for evaluating CWE-level vulnerability prediction in Python and proposed that the penalty could theoretically also serve as a training signal. This paper tests that proposal empirically, comparing three delivery mechanisms–supervised fine-tuning, a dual-head classification loss, and reinforcement learning with a dense reward derived from the normalised penalty. We find that supervised approaches consistently regress below the zero-shot baseline under distribution shift, while GRPO succeeds and generalises across independent training seeds. Using a default, non-test-informed configuration, GRPO reduces the mean cumulative ALPHA penalty of Qwen2.5-Coder-7B-Instruct on Security Hardening and Adversarial Testing (SVEN) by 13.5%. A further test-selected configuration reduces the mean penalty by 22.9% (27.9% in the best-observed run), reaching statistical parity with its 4.5× larger zero-shot teacher. Our experiments show that reinforcement learning outperforms the alternative delivery mechanisms we test. We conclude that delivery mechanism, rather than the specific hierarchical structure of the reward, is the primary variable—though whether taxonomy-aware structure contributes further remains an open question.