DashAct is introduced, to the authors' knowledge the first benchmark to diagnose failures at a fine-grained level within the same dashboard task, and measures the minimum support an agent needs to recover rather than scoring isolated skills.
Abstract
Interactive dashboards require users to reveal and connect evidence across stateful interactions. Although graphical user interface (GUI) agents could automate this process, existing dashboard benchmarks primarily report final answers or task success. They provide limited insight into whether failures arise from maintaining the analytical process, selecting actions, or grounding visual targets. We introduce DashAct, to our knowledge the first benchmark to diagnose these failures at a fine-grained level within the same dashboard task. DashAct contains 357 human-verified interaction trajectories with milestone dependencies and hierarchical target annotations. Its progressive diagnostic cascade evaluates end-to-end execution, restores verified context for next-action prediction, and provides target semantics and a local view for visual grounding. By progressively restoring the conditions for success, DashAct measures the minimum support an agent needs to recover rather than scoring isolated skills. Experiments show that current models struggle even as support is added. The cascade outcomes reveal bottlenecks hidden by end-to-end scores and provide actionable guidance for improving GUI agents.
Evidence-First Reflection (EFR), a two-stage reflector that explicitly decouples action-induced visual differences extraction from outcome verification, makes reflection better grounded in screen transitions, while reducing both visual search complexity and reasoning burden.
Yijie Ma, Chao-Yue Niu, Fan Wu et al.· 0 citations
This work introduces Living Dashboards, whose views are born, wither, revive, and die in response to how they are used, and formalizes the concept as a four-dimensional design space and implements it in Living Dashboard, a web-based prototype.
Ming-Yu An, Heyon Jeon, Sungbok Shin et al.· 0 citations
Understanding where and why Graphical User Interface (GUI) agents fail is essential for building more reliable systems, yet current evaluation relies on step accuracy, a metric that treats each screen independently and overlooks the underlying structure of GUI environments. This leads to two critical blind spots: (1) f...
Shao-Qing Zhang, Ke-Hai Chen, Xue-Feng Bai et al.· 0 citations
This work introduces WidgetVA, a widget-centric agentic VA framework that standardizes interactive components as structured widgets with unified action and perception-query APIs and presents WidgetVABench, a benchmark of single- and multi-widget VA tasks that require agents to perform multi-step interactions to uncover...
Yu-Tong Chen, Zhi-Ke Tang, Zhi-Hao Mai et al.· arXiv.org· 0 citations
MUSE is presented, an interactive meta-agent that enhances user understanding and control of agentic data science systems by dynamically restructuring low-level execution traces into multiple semantic levels that support navigation from high-level overviews to low-level implementation details.
Wei-Hao Chen, Weixi Tong, Yuan Tian et al.· 0 citations
UI-Venus-2 is presented, a general-purpose foundation GUI agent designed to operate across mobile, web, and desktop environments through a unified closed-loop reasoning-action framework that integrates safety-aware mechanisms to ensure controlled execution of consequential actions.
Venus Team, Zhuo-Hang Cai, Hao-Xin Chen et al.· 4 citations
Known for his clear and elegant writing style, Bertsekas shaped fields from control and optimization to large-scale computation and artificial intelligence.
MIT News · Artificial Intelligence· news.mit.eduJul 7, 2026
The professor of physics and inaugural director of the NSF AI Institute for Artificial Intelligence and Fundamental Interactions will lead LNS and continue his research in particle physics.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.