Hong-Liu Cao

Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error

Auditing two $\tau^2$-bench domains, a taxonomy of policy loopholes is developed and it is shown that affected tasks produce unreliable scores: they lower scores across different models in different ways and make every model less consistent across repeated trials.