TL;DR
This work introduces PolyWorkBench, a benchmark for evaluating LLM agents on multilingual long-horizon workplace workflows, and proposes a hybrid framework that combines structural grading, executable verification, and LLM-based semantic assessment to enable comprehensive evaluation.
WorldBench is presented: a comprehensive, multilingual benchmark of genuine, persona-grounded everyday workflows, where agents can act in a sandbox via structured actions, and Constrained Task Success (CTS), which combines natural language instructions and testbeds to score task completion, minimal modification, and ot...
OmnilingualGAIA2 is introduced, a machine-translated expansion of the GAIA2 agentic benchmark, covering ten target languages spanning five writing systems, paired with a localised and human-calibrated multilingual verifier, and it is argued that multilingual agentic evaluation must become a standard part of the reporti...
OmniaBench provides a broad and diagnostic benchmark for characterizing the capability boundaries of general agents across diverse scenarios with explicit state spaces, and introduces a ten-dimensional capability taxonomy and eight compositional atomic difficulty factors to support fine-grained evaluation and analysis.
This work reveals that retrieval from a structured agent repository provides a cost-efficient, accurate, and controllable alternative to dynamic agent generation, responding to the strict demands of industrial applications.
This work systematically study AI products with demonstrated adoption, together with their product workflows and users, to identify real-world tasks for which AI has established practical demand across diverse professional domains and establishes StartupBench as an empirical measure of progress toward E2E completions o...
SQBench is introduced, a benchmark for evaluating production-oriented task delivery by language-model agents and shows that functional completion alone does not fully characterize delivery quality and that risk determinations should be reported separately.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
