Evaluating open-ended large language model responses remains difficult because response quality depends not only on factual correctness and task completion, but also on subjective and scenario-dependent user experience factors. Existing benchmarks and automatic evaluators are effective for coarse-grained assessment, yet often provide limited insight into how different types of quality failures affect user-perceived response quality. To address this gap, we propose AutoQABench, an initial three-level user experience benchmark for automated evaluation of open-ended LLM responses. The benchmark decomposes response quality into three progressively organized levels: basic acceptability constraints, scenario-specific task effectiveness, and preference-sensitive experiential quality. Based on this design, we construct a dataset covering representative scenarios, including summarization, elaboration, emotional support, and responses to misleading premises, together with expert-defined evaluation standards. We also develop an LLM-based modular evaluator that performs stepwise assessment and generates both intermediate judgments and a final rating. Experimental results show that AutoQABench achieves good agreement with expert judgments and supports analysis at the module, scenario, and final-grade levels. Rather than replacing human evaluation, AutoQABench is intended as a scalable auxiliary evaluator that helps identify overlooked risks, task-completion failures, and quality differences in open-ended LLM interactions.
Tianyou Wang, Fei Yuan, Chia-Ju Miao et al.· International Conference on...· 0 citations
Language model agents now execute bounded tasks reliably. Whether they can sustain effective decision-making over long horizons, where actions have cumulative consequences and the environment responds to their choices, remains largely unmeasured. FM-Bench (Football Management Benchmark) measures this. An LLM agent runs a football club for 20 in-game years through 26 tools and roughly 340 to 400 decision stops. It drafts a squad on the same budget as every rival, trades players, negotiates contracts, invests in facilities and youth, sets lineups, and answers to a board that can fire it, while a deterministic engine accumulates every year into one final score with no LLM judge or human rater. The solo track plays each of 15 frontier models against a frozen scripted world, and the Arena places the same models plus a scripted anchor in one shared 20-year world; to our knowledge, the first head-to-head evaluation at this scale. We measure six behavioral capabilities behind the score. Across three seeds, all 15 models complete every horizon while the blind scripted baselines die out in most of theirs, and claude-fable-5 tops the solo board on mean score and the Arena, where the title nonetheless rotates among ten models. Neither scale, price, nor vendor predicts the order; the order settles only late in the horizon, and the best first-play human lands only at the bottom of the model board. What separates the models is managerial behavior rather than computation. Higher-scoring models reduce slow-payoff investment near the end, keep cash invested rather than idle, and open renewals well before the deadline, while token spend predicts nothing. No model learns the market's hidden prices from hundreds of rejected bids, and self-managed memory fails in two opposite modes: an archive that only grows or a plan rewritten every season. Code is available at https://github.com/Analogy-AI/fm-bench.
Tianyou Wang, Chongyang Gao, Kezhen Chen et al.· 0 citations