TL;DR
A Au-diting black-box (behavioral) detection of un-faithful CoT against FaithCoT-Bench’s human annotations, and independently verify and resolve a documentation–data mismatch in the bench-mark’s label semantics.
Auditing black-box (behavioral) detection of unfaithful CoT against FaithCoT-Bench's human annotations, the authors find answer correctness structures the problem at every level.
The analysis shows that CoTs do not reliably track visual evidence that influences model predictions, and it is found that Predict-then-Explain explanations align more strongly with perturbation-induced probability shifts than pre-answer CoTs, while binary vCT scores are often nearly saturated.
Grounded language-model systems are often evaluated by final answer accuracy, yet a correct answer can be unsupported, drawn from the wrong source, or produced when evidence is insufficient or contradictory. We introduce EviScope, a paired counterfactual benchmark that holds the question fixed while adding, removing, d...
Chain-of-thought (CoT) reasoning traces are increasingly proposed as a mechanism for AI oversight: a monitor inspecting a model's reasoning can, in principle, detect misbehavior invisible from outputs alone. This assumes CoT surfaces what a model is instructed to do regardless of the instructions given. We test this as...
While steering reliably increases cue acknowledgment for only the largest model, it is found that when steering is effective, its effect generalizes broadly across cue types and datasets--in cross-cue and cross-dataset analyses, effect size is determined primarily by the evaluation setting, rather than the vector's tra...
This work compares monitors across reasoning modes, from explicit CoT to weakly- and strongly-supervised latent CoT, on math reasoning and question answering and finds that, in this setup, monitorability depends more on properties of the task and the level of access to model internals than on the reasoning mode.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
