Evaluating Open-Weight E-Commerce Agents with Environment-Grounded Verification

• URL: http://arxiv.org/abs/2609.16093v1
• Date: Mon, 14 Sep 2026 13:35:32 GMT
• Title: Evaluating Open-Weight E-Commerce Agents with Environment-Grounded Verification
• Abstract summary: A shopping conversation has many routes to the same cart, and a task-success rate reduces all of them to one score.<n>We build a deterministic and reproducible e-commerce environment that precommits each trial's customer and trajectory parameters.<n>Across eight open-weight agents from 20B to 35B parameters, with 160 trials per agent and 44 metrics, the resulting capability profiles distinguish under-action, over-purchase, unsupported product attributes, and poor search.
• Score: 5.646213674986304
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A shopping conversation has many routes to the same cart, and a task-success rate reduces all of them to one score. We build a deterministic and reproducible e-commerce environment that precommits each trial's customer and trajectory parameters, including the persona, difficulty, target cart, and an item reveal schedule. A simulated consumer attempts to buy a target cart from the environment with assistance from the evaluated model. The environment guides the simulator's actions and records every assistant action alongside the environment state at that point. After the trial, these records allow the evaluator to assess individual parts of the conversation against the retained evidence. For example, the evaluator penalizes a search for failing to surface a target product only when the customer has already mentioned that product. We further use this evidence to apply different penalties to tool calls depending on how the assistant's actions compare with an expected tool-call set. Our environment also interacts with the simulator bidirectionally, reading its output to stop the trial when the simulator determines that the customer has become too frustrated and injecting directives in real time that specify when to explore, defer buying an item, or recall a previous exchange. This interaction creates an open-ended and verifiable simulation. Across eight open-weight agents from 20B to 35B parameters, with 160 trials per agent and 44 metrics, the resulting capability profiles distinguish under-action, over-purchase, unsupported product attributes, and poor search, all of which terminal success obscures.

Related papers

• Towards Faithful Simulation of Human Shopping Behavior [57.234936207923]
  We present RecVerse, a GUI-grounded simulation agent that perceives pages through screenshots and produces faithful multi-turn trajectories.<n>For the memory challenge, RecVerse adopts a cognitive-inspired hierarchical memory: Working Memory for short-term focus, Episodic Memory for in-session traces, and Preference Memory for high-level intent.<n>For the optimization challenge, RecVerse is optimized with a trajectory-level objective that scores entire sessions.
  arXiv  Detail & Related papers  (2026-08-21T03:24:32Z)
• ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step [74.83535434786145]
  We introduce ScrambleToolBench, an interactive terminal benchmark designed to isolate behavioral reasoning.<n>The benchmark requires agents to uncover hidden tool behaviors entirely through trial-and-error interaction.<n>Our evaluation of state-of-the-art language models reveals that successful initial discovery does not translate into robust adaptation.
  arXiv  Detail & Related papers  (2026-08-03T15:07:46Z)
• BehaviorBench: Modeling Real-World User Decisions from Behavioral Traces [53.55407626944686]
  textscBehaviorBench is a benchmark for evaluating personalized decision modeling from real-world behavioral traces.<n>textscBehaviorBench reconstructs wallet-level decision histories from observed public prediction-market and on-chain records.<n>Across 2,000 evaluation wallets, the benchmark contains 141,445 Belief instances and 1,485,972 Trade instances.
  arXiv  Detail & Related papers  (2026-06-01T19:04:36Z)
• SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agents [8.496158383333999]
  SimGym is a framework for simulating A/B tests on e-commerce storefronts using vision-structured model (VLM) agents operating in a live browser.<n>It reduces experimental cycles from weeks to under an hour, enabling rapid experimentation without exposing real buyers to candidate variants.
  arXiv  Detail & Related papers  (2026-05-19T00:46:41Z)
• KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation [72.01173512175531]
  KnowU-Bench is an online benchmark for personalized mobile agents.<n>It covers 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks.<n>Our experiments reveal a striking degradation: agents that excel at explicit task execution fall below 50% under vague instructions.
  arXiv  Detail & Related papers  (2026-04-09T16:50:50Z)
• Analyzing the Shopping Journey: Computing Shelf Browsing Visits in a Physical Retail Store [0.5161531917413708]
  We introduce an algorithm that computes shoppers' shelf visits''<n> Shelf visits are extracted from trajectories obtained via machine vision-based 3D tracking and overhead cameras.<n>An analysis of the results shows that the algorithm can recognize customers' browsing activity when evaluated in an environment different from the one on which calibration was performed.
  arXiv  Detail & Related papers  (2026-01-02T01:40:12Z)
• SessionIntentBench: A Multi-task Inter-session Intention-shift Modeling Benchmark for E-commerce Customer Behavior Understanding [64.45047674586671]
  We introduce the concept of an intention tree and propose a dataset curation pipeline.<n>We construct a sibling multimodal benchmark, SessionIntentBench, that evaluates L(V)LMs' capability on understanding inter-session intention shift.<n>With 1,952,177 intention entries, 1,132,145 session intention trajectories, and 13,003,664 available tasks mined using 10,905 sessions, we provide a scalable way to exploit the existing session data.
  arXiv  Detail & Related papers  (2025-07-27T09:04:17Z)
• AutoAssign: Differentiable Label Assignment for Dense Object Detection [94.24431503373884]
  Auto COCO is an anchor-free detector for object detection. It achieves appearance-aware through a fully differentiable weighting mechanism. Our best model achieves 52.1% AP, outperforming all existing one-stage detectors.
  arXiv  Detail & Related papers  (2020-07-07T14:32:21Z)
• Counter-example Guided Learning of Bounds on Environment Behavior [11.357397596759172]
  We present a data-driven solution that allows for a system to be evaluated for specification conformance without an accurate model of the environment. Our approach involves learning a conservative reactive bound of the environment's behavior using data and specification of the system's desired behavior.
  arXiv  Detail & Related papers  (2020-01-20T19:58:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.