Skip to content

The Hard Part Comes After Search: Benchmarking Web Agents on Synthesizing, Organizing, and Displaying Knowledge

Sep 2026 · Conference on Empirical Methods in Natural Language Processing · Vol abs/2609.30604 · 1 citation · 72 references
Computer Science

Abstract

Existing computer-use agent benchmarks do not fully evaluate agents acting as assistants. A useful assistant retrieves information across complex, multi-step workflows, synthesizes it into artifacts (documents, presentations, spreadsheets), and navigates program interfaces to produce a coherent final product. Such workflows demand reasoning and synthesis, decomposition of complex tasks, as well as visual and spatial understanding. To study agents on workflows like these, we introduce KNOWS, a benchmark of open-ended, complex, browser-based tasks that jointly evaluate these capabilities, with each task culminating in a produced artifact. To write tasks, we develop a task design rubric and a protocol for ensuring that tasks meet the requirements. Each task is paired with an evaluator, a program that combines deterministic checks with LLM judgments to balance the richness, reliability, and automation tradeoff inherent to agent evaluation. We evaluate and analyze frontier computer-use agents and browser-based harnesses. They achieve moderate scores on partial-success metrics, but the best performer fully succeeds in fewer than 3% of our complex, long-horizon tasks. Failures on visual steps render the resulting artifacts unusable, even when agents complete more than 50% of other evaluation steps. Our results expose limitations of current agents acting as end-to-end assistants, and call for progress on tool use, visual understanding, and long-horizon reasoning.

Read PDF

Similar papers

Review Open access 2026

How Experienced Developers Can Get More Value from Agents

AI agents are becoming a fundamental part of modern software creation, helping developers in generating code, debugging, designing systems, etc. But there is a clear difference between how beginners and experienced software engineers get benefits from these tools. Newbies usually depend on agents for one-time prompts a...

Madhurima Kommuru, Srujana Pulipaka · 0 citations
#artificial intelligence Preprint Sep 2026

DAREBench: Deployment-Aware and Reliable Evaluation of Models as Agents

DAREBench (Deployment-Aware and Reliable Evaluation of Models as Agents), a benchmark designed to capture workload variation and support reliable agent evaluation, is introduced, suggesting that agent deployment and model selection should consider workload profiles, deployment mode, and accuracy--cost trade-offs rather...

Yu Liu, Zhi-Lin Liu, Zhi-Wei Yang et al. · 0 citations
Preprint Oct 2026

OOPMAS: Object-Oriented Multi-Agent Systems for Query-Level Workflow Generation

OOPMAS is introduced, a training-free framework that generates both the agent set and the coordination workflow at the granularity of individual queries that achieves 89.6% accuracy on a mixed-task benchmark of queries spanning code, math, and QA.

Qi Cheng, Sheng-Yu Chen, Wei Cheng et al. · 0 citations
Review Aug 2026

MUSE: An Interactive Meta-Agent for Understanding and Steering LLM-powered Data Science Systems

MUSE is presented, an interactive meta-agent that enhances user understanding and control of agentic data science systems by dynamically restructuring low-level execution traces into multiple semantic levels that support navigation from high-level overviews to low-level implementation details.

Wei-Hao Chen, Weixi Tong, Yuan Tian et al. · 0 citations

CUA-SWE: When Computer-Use Agents Meet Visual Software Engineering

This evaluation characterizes how frontier agents combine source-level execution with application screenshots and graphical interaction to produce verified software changes, and examines performance across domains and task information requirements, alongside the development behaviors associated with successful repairs.

P. Wang, Chen-Hao Liang, Ze-Long Xu et al. · 2 citations
#artificial intelligence Preprint Oct 2026

TaReD: Tool-Aware Recursive Decomposition for Long-Horizon Tasks

Agents combine reasoning with tools to interact with external systems and complete real-world tasks. Early agents typically interleave reasoning and actions along a single execution chain. On complex tasks, this chain becomes unreliable because growing histories obscure intermediate dependencies and allow early plannin...

Wei-Hao Mao, Zhi-Kai Chen, De-Chuan Zhan et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.