Skip to content

Evaluating Open-Weight E-Commerce Agents with Environment-Grounded Verification

Sep 2026 · arXiv.org · Vol abs/2609.16093 · 0 citations · 30 references
Computer Science

TL;DR

A deterministic and reproducible e-commerce environment is built that precommits each trial's customer and trajectory parameters, including the persona, difficulty, target cart, and an item reveal schedule, and the resulting capability profiles distinguish under-action, over-purchase, unsupported product attributes, and poor search.

Abstract

A shopping conversation has many routes to the same cart, and a task-success rate reduces all of them to one score. We build a deterministic and reproducible e-commerce environment that precommits each trial's customer and trajectory parameters, including the persona, difficulty, target cart, and an item reveal schedule. A simulated consumer attempts to buy a target cart from the environment with assistance from the evaluated model. The environment guides the simulator's actions and records every assistant action alongside the environment state at that point. After the trial, these records allow the evaluator to assess individual parts of the conversation against the retained evidence. For example, the evaluator penalizes a search for failing to surface a target product only when the customer has already mentioned that product. We further use this evidence to apply different penalties to tool calls depending on how the assistant's actions compare with an expected tool-call set. Our environment also interacts with the simulator bidirectionally, reading its output to stop the trial when the simulator determines that the customer has become too frustrated and injecting directives in real time that specify when to explore, defer buying an item, or recall a previous exchange. This interaction creates an open-ended and verifiable simulation. Across eight open-weight agents from 20B to 35B parameters, with 160 trials per agent and 44 metrics, the resulting capability profiles distinguish under-action, over-purchase, unsupported product attributes, and poor search, all of which terminal success obscures.

View source

Similar papers

#artificial intelligence Open access Sep 2026

RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce

This work introduces REALWORLDSHOP, a benchmark built on 3.28M grounded products, structured shopping episodes, a profile-grounded and actioncontrolled user simulator, and role-play evaluation, and proposes REALSHOP_AGENT, an executable session-control framework with explicit state management, shopping-flow control, ca...

Xin-Wei Yang, Ke-Long Mao, Yu-Dong Guo et al. · 1 citation
#artificial intelligence Preprint Oct 2026

What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worlds

Effective social interaction requires more than fluent responses: agents must reach plans that fit the participants'needs and circumstances. Existing evaluations of social agents mainly rely on LLM judges that rate the conversation, without checking the resulting plan. We introduce HICO-Bench, a benchmark of 180 everyd...

Cai-Xin Kang, Ke Wang, Yu-Chuan Wu et al. · 1 citation
#artificial intelligence Preprint Sep 2026

ERPBench: A State-Grounded Evaluation Paradigm for Computer-Use Agents in Enterprise Software

ERPBench is introduced, a benchmark that evaluates screenshot-only agents on a live and reproducible system and scores each task against ground-truth values in its database and presents a production-grade harness that gates agent actions behind human approval for safe deployment.

Kratika Bhagtani, K. Sridhar, M. B. Pouyan et al. · 1 citation

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.