J4U is proposed, a jailbreak-derived technique for UQ that empirically reproduces the behavioral signatures predicted by the relaxation theory and provides a practical tool for UQ in black-box LRM deployment.
Abstract
While Large Reasoning Models (LRMs) excel at complex reasoning, alignment through reinforcement learning often induces systemic overconfidence. In production environments, where logits may be unavailable, robust black-box uncertainty quantification (UQ) is essential for trustworthiness and safety. Focusing on question-answering for LRMs, we show that existing black-box methods, such as paraphrase-based self-consistency and confidence verbalization, offer little to no improvement over simple repeated sampling, suggesting that alignment suppresses useful output variability. We introduce prompt-level relaxation operators that broaden the model's effective output distribution by approximating the effect of an optimal policy obtained with a stronger KL-regularization parameter, hence closer to the reference model. Theoretically, we demonstrate that relaxation improves calibration. We propose Jailbreak for Uncertainty (J4U), a jailbreak-derived technique for UQ that empirically reproduces the behavioral signatures predicted by our relaxation theory. Across 3 datasets and 4 LRMs, including a closed-source production model, J4U's improvement over repeated sampling achieves statistical significance in up to 6 times more LRM-dataset-metric settings than the strongest black-box UQ state-of-the-art baseline we evaluate, with average ECE reductions up to 5 times larger. These results provide a practical tool for UQ in black-box LRM deployment.
It is revealed that off-the-shelf LRMs exhibit a confidence prior heavily concentrated on a few high values, which persists throughout RL, and it is proved that this concentration suppresses policy gradient updates for rarely sampled confidence values and inflates the lower bound on expected Brier risk.
This work introduces DSAR (Deceptive Safety Alignment Rate), a metric that jointly assesses reasoning traces and final answers to quantify their safety inconsistency, and proposes SARA (Safety-Aware Reasoning Alignment), an RL-based method that rewards both safety-aware reasoning and safe final answers.
Xiang-Yu Zhou, Saleh Zare Zade, Rafi Ibn Sultan et al.· arXiv.org· 0 citations
Recent methods for Reinforcement Learning with Verifiable Rewards (RLVR) use rollout diversity to improve reasoning performance or training efficiency. Despite these gains, the extent to which differences among correct-answer rollouts reflect useful alternative reasoning or unresolved errors remains poorly understood....
Sohyun Son, Yerim Hong, Jaeyun Park et al.· 1 citation
Experiments show that CALM reduces token consumption by 18.6%-25.0% compared with full CoT reasoning while maintaining performance, and can even improve accuracy by 0.07%-1.48% by reducing redundant reasoning.
Sheng-Jing Liu, Fei-Yang Li, Yuxuan Yang et al.· 0 citations
Reinforcement learning with verifiable rewards (RLVR) can enable generalizable improvement for large language model (LLM) reasoning. This paper studies how to incentivize such generalization via reasoning strategies. We study this question with a controllable equation-solving framework: finding multiple inputs x for eq...
BayesianPRM is proposed, which treats the process reward as a random variable within a reward hypothesis space and maintains a Bayesian belief over it, and yields a reliability- and conservatism-aware Bayesian belief, whose expectation determines the process reward.
Zhong-Jian Qiao, Zhi-Xiang Zhou, Jiafei Lyu et al.· 1 citation
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Machine-learning networks can learn without detailed instructions from a designer. Those that do may offer insight into how the brain reorganizes itself during learning.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.