A deterministic and reproducible e-commerce environment is built that precommits each trial's customer and trajectory parameters, including the persona, difficulty, target cart, and an item reveal schedule, and the resulting capability profiles distinguish under-action, over-purchase, unsupported product attributes, and poor search.
Abstract
A shopping conversation has many routes to the same cart, and a task-success rate reduces all of them to one score. We build a deterministic and reproducible e-commerce environment that precommits each trial's customer and trajectory parameters, including the persona, difficulty, target cart, and an item reveal schedule. A simulated consumer attempts to buy a target cart from the environment with assistance from the evaluated model. The environment guides the simulator's actions and records every assistant action alongside the environment state at that point. After the trial, these records allow the evaluator to assess individual parts of the conversation against the retained evidence. For example, the evaluator penalizes a search for failing to surface a target product only when the customer has already mentioned that product. We further use this evidence to apply different penalties to tool calls depending on how the assistant's actions compare with an expected tool-call set. Our environment also interacts with the simulator bidirectionally, reading its output to stop the trial when the simulator determines that the customer has become too frustrated and injecting directives in real time that specify when to explore, defer buying an item, or recall a previous exchange. This interaction creates an open-ended and verifiable simulation. Across eight open-weight agents from 20B to 35B parameters, with 160 trials per agent and 44 metrics, the resulting capability profiles distinguish under-action, over-purchase, unsupported product attributes, and poor search, all of which terminal success obscures.
A framework for evaluating and improving a large-scale, multi-agent shopping assistant in production in production is presented, and lessons from its use are reported.
Kasra Hosseini, Wen-Sen Cheng, Marco-Andrea Buchmann et al.· 0 citations
This work introduces REALWORLDSHOP, a benchmark built on 3.28M grounded products, structured shopping episodes, a profile-grounded and actioncontrolled user simulator, and role-play evaluation, and proposes REALSHOP_AGENT, an executable session-control framework with explicit state management, shopping-flow control, ca...
Xin-Wei Yang, Ke-Long Mao, Yu-Dong Guo et al.· Conference on Empirical Meth...· 1 citation
Effective social interaction requires more than fluent responses: agents must reach plans that fit the participants'needs and circumstances. Existing evaluations of social agents mainly rely on LLM judges that rate the conversation, without checking the resulting plan. We introduce HICO-Bench, a benchmark of 180 everyd...
Cai-Xin Kang, Ke Wang, Yu-Chuan Wu et al.· 1 citation
The Article carries the argument from the browser to the sidewalk, where embodied robots will soon shop for people who cannot, and shows that the Department of Justice's refusal to let public accommodations ban classes of new mobility technology supplies the method.
This work introduces StoreBench, a live-commerce environment in which an agent runs a mid-size online apparel store on a production-grade commerce backend, testing long-horizon planning and economic judgment under uncertainty.
Daksh Raghuvanshi, Ved Vedere, Yi-Fan Wang· 1 citation
ERPBench is introduced, a benchmark that evaluates screenshot-only agents on a live and reproducible system and scores each task against ground-truth values in its database and presents a production-grade harness that gates agent actions behind human approval for safe deployment.
Kratika Bhagtani, K. Sridhar, M. B. Pouyan et al.· 1 citation
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 17, 2026
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026