Affora is presented, a design system that supports both readers and agents while preserving visual freedom and familiar human workflows through a shared interface rather than a separate agent-only surface.
Abstract
Computer-use agents increasingly operate software designed for people, but interfaces often leave actions or task state unclear to machine readers. We present Affora, a design system that supports both readers while preserving visual freedom and familiar human workflows. Three controlled studies examine component implementations, visual variation, and interaction-design principles. Their findings inform guidance from individual components to complete sites, supported by reusable implementations and executable checks. Agent performance depends on the interaction meaning available through its interface representation; substantial visual variation remains possible when that meaning is preserved. Evaluation on independently authored interfaces shows gains where Affora addresses existing deficits, but limited effects where those deficits are absent or outside its coverage. A workflow case provides preliminary evidence of reduced interaction cost. Affora connects user experience and agent experience through a shared interface rather than a separate agent-only surface.
A three-week diary study with 8 blind users using OLLA, a screen-reader-accessible CUA prototype, collecting 1,258 commands across 12 applications with screenshots, UI trees, model responses, and action traces reveals grounding, planning, constraint-tracking, and termination failures, while interviews reveal beyond-aut...
Satwik Ram Kodandaram, Monalika Padma Reddy, Xiaojun Bi et al.· 0 citations
Advances in multimodal large language models have enabled mobile GUI agents that can interpret screen content and autonomously perform tasks across applications. While prior research has largely focused on task completion and agent autonomy, less attention has been paid to how users can remain involved in the delegatio...
Ching-Ting Lin, Zhi-Hong Ye, Yao-Jen Cheng et al.· Proceedings of the 28th Inte...· 0 citations
This work implements ABRA, a framework-agnostic prototype of Agent-Bound UI, a framework-agnostic prototype of Agent-Bound UI in which users and agents collaborate through chat and shared interactive canvases, and evaluates four LLM-backed agents on canvas authoring, owner-side consequence resolution, and cross-agent i...
The prevailing approach to computer-use agents couples a model with a domain-specific harness: a browser or desktop environment equipped with human engineered tools that are fixed before task execution. As models'coding capabilities improve, the GUI native and static harness prevents them from direct programmatic opera...
Ya-Dong Lu, Theo Lee, Yi-Fei Li et al.· 0 citations
Large language model agents are emerging as an important new class of users of desktop applications. However, existing desktop interfaces were designed for human end users or application developers, leaving agents to interpret visual controls or discover and compose low-level software operations. This mismatch can hind...
Computer-use agents are limited by the interfaces humans give them. An agent may type a spreadsheet cell by cell when a single line of code would do, or print all two thousand lines of a file through a shell to find one function. Both the interfaces and the knowledge of when to use each are built by hand, which cannot...
S. Agashe, Yuzhe Yang, Gonzalo Gonzalez-Pumariega et al.· 0 citations
Known for his clear and elegant writing style, Bertsekas shaped fields from control and optimization to large-scale computation and artificial intelligence.
MIT News · Artificial Intelligence· news.mit.eduJul 7, 2026
The professor of physics and inaugural director of the NSF AI Institute for Artificial Intelligence and Fundamental Interactions will lead LNS and continue his research in particle physics.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
