ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents

• URL: http://arxiv.org/abs/2508.04266v1
• Date: Wed, 06 Aug 2025 09:51:30 GMT
• Title: ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents
• Abstract summary: Existing benchmarks in e-commerce primarily focus on basic user intents, such as finding or purchasing products.<n>We propose ShoppingBench, a novel end-to-end shopping benchmark designed to encompass increasingly challenging levels of grounded intent.<n>We provide a large-scale shopping sandbox that serves as an interactive simulated environment, incorporating over 2.5 million real-world products.
• Score: 21.399546417224755
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing benchmarks in e-commerce primarily focus on basic user intents, such as finding or purchasing products. However, real-world users often pursue more complex goals, such as applying vouchers, managing budgets, and finding multi-products seller. To bridge this gap, we propose ShoppingBench, a novel end-to-end shopping benchmark designed to encompass increasingly challenging levels of grounded intent. Specifically, we propose a scalable framework to simulate user instructions based on various intents derived from sampled real-world products. To facilitate consistent and reliable evaluations, we provide a large-scale shopping sandbox that serves as an interactive simulated environment, incorporating over 2.5 million real-world products. Experimental results demonstrate that even state-of-the-art language agents (such as GPT-4.1) achieve absolute success rates under 50% on our benchmark tasks, highlighting the significant challenges posed by our ShoppingBench. In addition, we propose a trajectory distillation strategy and leverage supervised fine-tuning, along with reinforcement learning on synthetic trajectories, to distill the capabilities of a large language agent into a smaller one. As a result, our trained agent achieves competitive performance compared to GPT-4.1.

Related papers

• SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks [73.92333717662558]
  SpatialWorld is a benchmark for evaluating the interactive spatial understanding of multimodal agents in complex real-world tasks.<n>It features 760 human-annotated tasks across diverse domains (e.g., household routines, travel, social collaboration)<n>For reliable evaluation, each task includes a human-validated initial state, a reference trajectory, and a terminal-state verifier.
  arXiv  Detail & Related papers  (2026-06-08T15:51:51Z)
• EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents [17.727481701114556]
  We introduce EgoBench, the first interactive multimodal benchmark for tool-using agents.<n>We implement a three-stage synergistic pipeline through which each task is designed to enforce the joint application of visual perception and tool-augmented multi-hop reasoning.<n>We additionally develop a multi-agent simulated user within EgoBench to evaluate agents' interaction capabilities, which generates high-fidelity, task-aligned responses.
  arXiv  Detail & Related papers  (2026-05-27T01:28:15Z)
• Shopping Companion: A Memory-Augmented LLM Agent for Real-World E-Commerce Tasks [6.67534912037121]
  LLM agents show promise for shopping tasks such as recommendations, budgeting, and bundle deals.<n>In this paper, we introduce a novel benchmark with a long-term memory setup, spanning two shopping tasks over 1.2 million real-world products.<n>We propose Shopping Companion, a unified framework that jointly tackles memory retrieval and shopping assistance while supporting user intervention.
  arXiv  Detail & Related papers  (2026-03-16T06:07:52Z)
• ChatShopBuddy: Towards Reliable Conversational Shopping Agents via Reinforcement Learning [88.87305027895657]
  This work investigates Reinforcement Learning-based optimization for shopping agents in real-world scenarios.<n>Agents must simultaneously satisfy multiple interdependent objectives spanning objective metrics (product correctness), subjective qualities (persuasiveness), outcome rewards (final response quality), and process rewards (tool efficiency)<n>Experiments demonstrate that our RL-trained agent, ChatShopBuddy, consistently outperforms larger models relying on generic reasoning.
  arXiv  Detail & Related papers  (2026-03-06T09:18:51Z)
• ShopSimulator: Evaluating and Exploring RL-Driven LLM Agent for Shopping Assistants [30.950450014358722]
  We introduce ShopSimulator, a large-scale and challenging Chinese shopping environment.<n>We evaluate LLMs across diverse scenarios, finding that even the best-performing models achieve less than 40% full-success rate.<n>Further training exploration provides practical guidance for overcoming these weaknesses.
  arXiv  Detail & Related papers  (2026-01-26T07:24:28Z)
• ShoppingComp: Are LLMs Really Ready for Your Shopping Cart? [21.021833054625095]
  ShoppingComp is a challenging real-world benchmark for rigorously evaluating LLM-powered shopping agents.<n>It introduces highly complex tasks under the principle of guaranteeing real products and ensuring easy verifiability.<n>The benchmark comprises 120 tasks and 1,026 scenarios, curated by 35 experts to reflect authentic shopping needs.
  arXiv  Detail & Related papers  (2025-11-28T08:32:54Z)
• How can we assess human-agent interactions? Case studies in software agent design [52.953425368394306]
  We make two major steps towards the rigorous assessment of human-agent interactions.<n>We propose PULSE, a framework for more efficient human-centric evaluation of agent designs.<n>We deploy the framework on a large-scale web platform built around the open-source software agent OpenHands.
  arXiv  Detail & Related papers  (2025-10-10T19:04:28Z)
• EcomScriptBench: A Multi-task Benchmark for E-commerce Script Planning via Step-wise Intention-Driven Product Association [83.4879773429742]
  This paper defines the task of E-commerce Script Planning (EcomScript) as three sequential subtasks.<n>We propose a novel framework that enables the scalable generation of product-enriched scripts by associating products with each step.<n>We construct the very first large-scale EcomScript dataset, EcomScriptBench, which includes 605,229 scripts sourced from 2.4 million products.
  arXiv  Detail & Related papers  (2025-05-21T07:21:38Z)
• EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents [63.43699771428243]
  EmbodiedBench is an extensive benchmark designed to evaluate vision-driven embodied agents.<n>We evaluated 24 leading proprietary and open-source MLLMs within EmbodiedBench.<n> MLLMs excel at high-level tasks but struggle with low-level manipulation, with the best model, GPT-4o, scoring only 28.9% on average.
  arXiv  Detail & Related papers  (2025-02-13T18:11:34Z)
• The BrowserGym Ecosystem for Web Agent Research [151.90034093362343]
  BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents.<n>We propose an extended BrowserGym-based ecosystem for web agent research, which unifies existing benchmarks from the literature.<n>We conduct the first large-scale, multi-benchmark web agent experiment and compare the performance of 6 state-of-the-art LLMs across 6 popular web agent benchmarks.
  arXiv  Detail & Related papers  (2024-12-06T23:43:59Z)
• SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation [89.24729958546168]
  Smartphone agents are increasingly important for helping users control devices efficiently.<n>We present SPA-Bench, a comprehensive SmartPhone Agent Benchmark designed to evaluate (M)LLM-based agents.
  arXiv  Detail & Related papers  (2024-10-19T17:28:48Z)
• UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling [22.885385107905222]
  We introduce UniBench, a unified implementation of 50+ vision-language model (VLM) benchmarks. We showcase the utility of UniBench for measuring progress by evaluating nearly 60 publicly available vision-language models. We also release an easy-to-run UniBench code-base with the full set of 50+ benchmarks and comparisons across 59 models as well as a distilled set of benchmarks that runs in 5 minutes on a single GPU.
  arXiv  Detail & Related papers  (2024-08-09T01:41:05Z)
• PUMGPT: A Large Vision-Language Model for Product Understanding [18.70740237744492]
  PumGPT is the first e-commerce specialized LVLM designed for multimodal product understanding tasks. Our experiments show that PumGPT outperforms five other open-source LVLMs and GPT-4V in product understanding tasks.
  arXiv  Detail & Related papers  (2023-08-18T14:01:37Z)
• Entity-Graph Enhanced Cross-Modal Pretraining for Instance-level Product Retrieval [152.3504607706575]
  This research aims to conduct weakly-supervised multi-modal instance-level product retrieval for fine-grained product categories. We first contribute the Product1M datasets, and define two real practical instance-level retrieval tasks. We exploit to train a more effective cross-modal model which is adaptively capable of incorporating key concept information from the multi-modal data.
  arXiv  Detail & Related papers  (2022-06-17T15:40:45Z)
• ItemSage: Learning Product Embeddings for Shopping Recommendations at Pinterest [60.841761065439414]
  At Pinterest, we build a single set of product embeddings called ItemSage to provide relevant recommendations in all shopping use cases. This approach has led to significant improvements in engagement and conversion metrics, while reducing both infrastructure and maintenance cost.
  arXiv  Detail & Related papers  (2022-05-24T02:28:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.