SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agents

• URL: http://arxiv.org/abs/2605.19219v1
• Date: Tue, 19 May 2026 00:46:41 GMT
• Title: SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agents
• Abstract summary: SimGym is a framework for simulating A/B tests on e-commerce storefronts using vision-structured model (VLM) agents operating in a live browser.<n>It reduces experimental cycles from weeks to under an hour, enabling rapid experimentation without exposing real buyers to candidate variants.
• Score: 8.496158383333999
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A/B testing remains the gold standard for evaluating modifications to e-commerce storefronts, yet it diverts traffic, requires weeks to reach statistical significance, and risks degrading user experience. We present SimGym, a framework for simulating A/B tests on e-commerce storefronts using vision-language model (VLM) agents operating in a live browser. The framework comprises three key components: (a) a traffic-grounded persona generation pipeline that derives per-shop buyer archetypes and intents from production clickstream data; (b) a live-browser agent architecture that combines multimodal perception over visual and browser-structured observations with episodic memory and guardrails to conduct coherent shopping sessions across control and treatment storefronts; and (c) an evaluation protocol that compares simulated outcome shifts with observed shifts in real buyer behavior. We validate SimGym on A/B tests of visually driven UI theme changes from a major e-commerce platform across diverse storefronts and product categories. Empirical results show that SimGym agents achieve strong agreement with observed outcome shifts, attaining 77% directional alignment with add-to-cart shifts observed across interface variants in real-buyer traffic. It reduces experimental cycles from weeks to under an hour, enabling rapid experimentation without exposing real buyers to candidate variants.

Related papers

• Evaluating Open-Weight E-Commerce Agents with Environment-Grounded Verification [5.646213674986304]
  A shopping conversation has many routes to the same cart, and a task-success rate reduces all of them to one score.<n>We build a deterministic and reproducible e-commerce environment that precommits each trial's customer and trajectory parameters.<n>Across eight open-weight agents from 20B to 35B parameters, with 160 trials per agent and 44 metrics, the resulting capability profiles distinguish under-action, over-purchase, unsupported product attributes, and poor search.
  arXiv  Detail & Related papers  (2026-09-14T13:35:32Z)
• Towards Faithful Simulation of Human Shopping Behavior [57.234936207923]
  We present RecVerse, a GUI-grounded simulation agent that perceives pages through screenshots and produces faithful multi-turn trajectories.<n>For the memory challenge, RecVerse adopts a cognitive-inspired hierarchical memory: Working Memory for short-term focus, Episodic Memory for in-session traces, and Preference Memory for high-level intent.<n>For the optimization challenge, RecVerse is optimized with a trajectory-level objective that scores entire sessions.
  arXiv  Detail & Related papers  (2026-08-21T03:24:32Z)
• ReactSim-Bench: Benchmarking Reactive Behavior World Model Simulation in Autonomous Driving [56.96697737229895]
  Reactive capability is a key property of data-driven behavior world model simulators for autonomous driving simulation systems.<n>We introduce ReactSim-Bench for evaluating the reactive capability of behavior world model simulation in autonomous driving.
  arXiv  Detail & Related papers  (2026-06-12T03:11:06Z)
• ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents [12.399936351655917]
  ShopGym is an integrated framework for realistic simulation and scalable benchmarking of e-commerce web agents.<n>ShopArena converts live storefronts into self-contained sandbox shops through shop specifications and a staged, validated generation process.<n>ShopGuru synthesizes benchmark tasks across seven skill categories, grounding each task in the shop's catalog, navigation structure, policies, and interaction affordances.
  arXiv  Detail & Related papers  (2026-05-15T16:00:38Z)
• SalesSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators [63.68151307455963]
  We present SalesSim, a framework and testbed for evaluating the ability of Multimodal Large Language Models (MLLMs) to simulate realistic, persona-driven customer behavior.<n>We measure consistency between the simulator's actions and its persona specifications, as well as conversational quality.<n>Our experiments demonstrate that UserGRPO boosts decision alignment of the baseline model by 13.8% while improving conversational quality.
  arXiv  Detail & Related papers  (2026-05-08T17:59:23Z)
• What Makes a Sale? Rethinking End-to-End Seller--Buyer Retail Dynamics with LLM Agents [22.408631930364788]
  We present RetailSim, an end-to-end retail simulation framework that models this pipeline in a unified environment.<n>We demonstrate its practical utility via decision-oriented use cases, including persona inference, seller-buyer interaction analysis, and sales strategy evaluation.
  arXiv  Detail & Related papers  (2026-04-06T06:38:35Z)
• SimAB: Simulating A/B Tests with Persona-Conditioned AI Agents for Rapid Design Evaluation [3.609531017498719]
  We present SimAB, a system that reframes A/B testing as a fast, privacy-preserving simulation using persona-conditioned AI agents.<n>Given design screenshots and a conversion goal, SimAB generates user personas, deploys them as agents that state their preference, aggregates results, and synthesizes rationales.
  arXiv  Detail & Related papers  (2026-03-01T10:08:27Z)
• SimGym: Traffic-Grounded Browser Agents for Offline A/B Testing in E-Commerce [8.496158383334]
  SimGym is a scalable system for rapid offline A/B testing using traffic-grounded synthetic buyers powered by Large Language Model agents operating in a live browser.<n>SimGym extracts per-shop buyer profiles and intents from production interaction data.<n>We validate SimGym against real human outcomes from real UI changes on a major e-commerce platform under confounder control.
  arXiv  Detail & Related papers  (2026-02-01T21:23:04Z)
• Exploring Recommender System Evaluation: A Multi-Modal User Agent Framework for A/B Testing [54.456400601801704]
  We introduce a multi-modal user agent for A/B testing (A/B Agent)<n>Specifically, we construct a recommendation sandbox environment for A/B testing, enabling multimodal and multi-page interactions.<n>We validated the potential of the agent as an alternative to traditional A/B testing from three perspectives: model, data, and features.
  arXiv  Detail & Related papers  (2026-01-08T03:33:43Z)
• See, Think, Act: Online Shopper Behavior Simulation with VLM Agents [58.92444959954643]
  This paper investigates the integration of visual information, specifically webpage screenshots, into behavior simulation via VLMs.<n>We employ SFT for joint action prediction and rationale generation, conditioning on the full interaction context.<n>To further enhance reasoning capabilities, we integrate RL with a hierarchical reward structure, scaled by a difficulty-aware factor.
  arXiv  Detail & Related papers  (2025-10-22T05:07:14Z)
• AgentA/B: Automated and Scalable Web A/BTesting with Interactive LLM Agents [35.8650712223701]
  A/B testing remains constrained by its dependence on the large-scale and live traffic of human participants.<n>We present AgentA/B, a novel system that automatically simulate user interaction behaviors with real webpages.<n>Our findings suggest AgentA/B can emulate human-like behavior patterns.
  arXiv  Detail & Related papers  (2025-04-13T21:10:56Z)
• TrafficBots: Towards World Models for Autonomous Driving Simulation and Motion Prediction [149.5716746789134]
  We show data-driven traffic simulation can be formulated as a world model. We present TrafficBots, a multi-agent policy built upon motion prediction and end-to-end driving. Experiments on the open motion dataset show TrafficBots can simulate realistic multi-agent behaviors.
  arXiv  Detail & Related papers  (2023-03-07T18:28:41Z)
• TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors [74.67698916175614]
  We propose TrafficSim, a multi-agent behavior model for realistic traffic simulation. In particular, we leverage an implicit latent variable model to parameterize a joint actor policy. We show TrafficSim generates significantly more realistic and diverse traffic scenarios as compared to a diverse set of baselines.
  arXiv  Detail & Related papers  (2021-01-17T00:29:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.