A benchmark score is a measurement instrument, yet most benchmarks read each item at a single canonical surface form. We ask whether that reading is faithful: when an item's intent is held fixed and only its meaning-preserving surface form varies, does the canonical-form score estimate model behavior well, and how much of any variation is decoding/judge noise rather than signal? We instantiate this in safety, a high-stakes setting with no gold label to average toward. To avoid prior confounds, we pre-author the reformulations (refusal-free, mostly non-LLM: machine back-translation and a Matrix-Language-Frame code-switch generator) so an identical surface form reaches every model, score all responses with one human-anchored, vendor-neutral judge (Claude, kappa = 0.86 vs. human on unsafe compliance, stable across languages, cross-checked by GPT-4o), and verify intent preservation. On 370 seeds x 5 surface forms x 5 models, no single transformation is uniformly most dangerous (6 of 20 per-transformation McNemar tests survive correction, most protective). Yet evaluating only the canonical prompt underestimates unsafe compliance: the union of unsafe outcomes across forms exceeds even the worst single form by 3.3-12.9 pp, with bootstrap 95% CIs excluding zero for all five models, and 5-13% of seeds safe on canonical are unsafe under some reformulation -- above a zero stochasticity floor (canonical resampled five times at temperature 0 gives 0/370 new exposures). The size of this gap is model-dependent (largest on Gemini 2.5 Pro). One form recovers only ~53% of a model's observed unsafe surface and about three reach 85% -- a redundancy characterization of this form set, not of a defined population. A benign control (XSTest) suggests the instability is bidirectional, though the benign and harmful pools are not item-matched. We release the dataset, code, and per-response labels.
Yongxin Zhou, Jun-Wei Yao, Yuanzhe Liu et al.· 1 citation
The AppraiSal benchmark is introduced, containing 996 emotional support conversations with human-annotated mental states, including salient cognitive appraisal dimensions, and PRISM, a multi-agent probabilistic framework grounded in Bayesian Inverse Planning, designed to improve LLMs'ability to identify context-specific appraisal dimensions.
Hainiu Xu, ZHAOYUE SUN, Hanqi Yan et al.· arXiv.org· 0 citations
This work introduces \textit{masked distillation}, a knowledge-distillation framework in which a student LLM is trained to predict only the solution tokens conditioned on the question, while a reasoning teacher provides feedback on the student's responses after conditioning on the question and its own CoT trace.
Agora is a framework that uses a confidence-calibrated auction to dynamically allocate tasks to expert models and tools, and treats reasoning steps as tradeable items, which bases allocation on calibrated competence rather than raw confidence.
Kaiji Zhou, A. Leonardis, Yue Feng· arXiv.org· 0 citations
Reach audiences
Advertise in front of researchers, engineers, and readers.
In Incognita, Incognita asks what happens when the task and success criterion stay fixed but access does not, and transforms eighteen customer-service tasks into three settings: direct access, one known intermediary, and six role-isolated participants whose capabilities must be discovered.
Inspired by dual-coding theory, this work proposes a memory architecture that uses parallel visual and verbal codes, which it calls DualMem, and views this as a step towards memory systems that preserve a richer record of agents'observations.
DART is introduced, a training-free routing framework that samples two cheap no-think drafts, accepts direct answering when the drafts agree, and predicts a thinking budget from draft entropy when they disagree, and preserves or improves always-thinking accuracy in most settings while reducing thinking-token use.
Jungseob Lee, Seongtae Hong, Seungjun Lee et al.· arXiv.org· 1 citation
It is argued that public tutoring benchmarks can better support positive-impact evaluation by reporting solving-oriented and pedagogy-oriented scores separately and by making disclosure-sensitive, student-agency-preserving criteria more explicit.
LALS (Latent Association Leaning Score) is introduced, a zero-shot metric that projects visual-token activations into the model's text-embedding space to measure concept associations per token and layer and shows that culturally loaded visual cues further modulate these internal associations.
Arnau Marin-Llobet, Simon Henniger, M. Banaji· arXiv.org· 1 citation
The conversational trajectories of ~12,000 randomly sampled Microsoft Bing Copilot users are analyzed and compared with data from WildChat-4.8M, finding that existing user behavior is difficult to change and the extent of user heterogeneity is demonstrated.
IntentVLA is introduced, a history-conditioned VLA framework that encodes recent visual observations into a compact short-horizon intent representation and uses it to condition chunk generation and improves rollout stability and outperforms strong VLA baselines.
Shijie Lian, Bin Yu, Xiaopeng Lin et al.· arXiv.org· 6 citations
A novel function hijacking attack (FHA) that manipulates the tool selection process of agentic models to force the invocation of an attacker-chosen function and is largely agnostic to the context semantics and remains effective across domains and function sets.
