E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation

• URL: http://arxiv.org/abs/2608.30730v1
• Date: Mon, 31 Aug 2026 13:03:57 GMT
• Title: E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation
• Abstract summary: We introduce E-Commerce Bench, the first open-source benchmark that integrates multi-round counterpart negotiation and dynamic events into a year-long business operation.<n>We evaluate 18 frontier models across seven dimensions, including year-end assets, and find that no single model dominates.
• Score: 54.50499240166176
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-horizon agentic tasks go beyond chaining short tasks over more interaction turns. Their evolving dynamic environments and long-range dependencies require Large Language Models (LLMs) to continually explore, learn from experience, and adapt their policies over thousands of steps. We introduce E-Commerce Bench, the first open-source benchmark that integrates multi-round counterpart negotiation and dynamic events into a year-long business operation. Over a 365-day year, an LLM agent concurrently runs multiple online stores, researching the market, negotiating with suppliers to source inventory, optimizing sales strategies, fulfilling orders, handling returns, and managing cash flow to maximize its end-of-year total assets. To construct a realistic merchant-side operating environment, the product and supplier data are derived from a real e-commerce platform, while a year-long calendar of promotions, natural disasters, and supply-chain shocks continually reshapes demand. For reproducibility, both sides of the market are deterministic: customer purchases and returns follow a fixed demand model, while a negotiation kernel determines supplier pricing, concessions, and decisions, with an LLM used only to verbalize them. We evaluate 18 frontier models across seven dimensions, including year-end assets, and find that no single model dominates. GPT-5.6 Sol earns the most, growing the 100,000 opening stake into 1,431,425, yet it ranks 16th of 18 on fraud avoidance and trails Fable5 in operational efficiency. Among open-weight models, Qwen3.8-Max-Preview leads with 416,252, 38% above GLM 5.2 (high), and achieves the strongest learning over the horizon, progressively bargaining down prices across repeated orders. Our code is available at https://github.com/QwenLM/E-CommerceBench.

Related papers

• MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations [18.706607921548326]
  Long-Term Coherence is the capacity to preserve purposeful behavior across extended horizons while adapting decisions to accumulated evidence.<n>Seller-side e-commerce provides a suitable setting for this evaluation through recurrent and interdependent decisions.<n> MerchantBench is a 365-day order-level simulation grounded in 98,843 real e-commerce product records and equipped with 26 tools for agent interaction.
  arXiv  Detail & Related papers  (2026-07-31T02:20:24Z)
• CEO-Bench: Can Agents Play the Long Game? [31.540356544517593]
  Language model agents are becoming proficient executors at isolated, short-horizon tasks.<n>Yet real-world challenges require a combination of sophisticated skills that remain largely untested in agents.<n>We introduce CEO-Bench, which evaluates these capabilities together by simulating a representative real-world task.
  arXiv  Detail & Related papers  (2026-06-16T23:37:52Z)
• $\ exttt{YC-Bench}$: Benchmarking AI Agents for Long-Term Planning and Consistent Execution [38.30265058791818]
  We introduce $texttYC-Bench$, a benchmark that evaluates capabilities by tasking an agent with running a simulated startup over a one-year horizon spanning hundreds of turns.<n>We evaluate 12 models, both proprietary and open source, across 3 seeds each.<n>Only three models consistently surpass the starting capital of $200K, with Claude Opus 4.6 achieving the highest average final funds at $1.27 M, followed by GLM-5 at $1.21 M at 11$times$ lower inference cost.
  arXiv  Detail & Related papers  (2026-04-01T17:52:19Z)
• Self-Improving World Modelling with Latent Actions [53.93276450137471]
  Internal modelling of the world is essential to reasoning and planning.<n>We propose SWIRL, a self-improvement framework that learns from state-only sequences.
  arXiv  Detail & Related papers  (2026-02-05T19:04:41Z)
• OneMall: One Architecture, More Scenarios -- End-to-End Generative Recommender Family at Kuaishou E-Commerce [68.7552227901176]
  OneMall is an end-to-end generative recommendation framework tailored for e-commerce services at Kuaishou.<n>It unifies the e-commerce's multiple item distribution scenarios, such as Product-card, short-video and live-streaming.<n>OneMall has been deployed, serving over 400 million daily active users at Kuaishou.
  arXiv  Detail & Related papers  (2026-01-29T14:22:39Z)
• xLAM: A Family of Large Action Models to Empower AI Agent Systems [111.5719694445345]
  We release xLAM, a series of large action models designed for AI agent tasks. xLAM consistently delivers exceptional performance across multiple agent ability benchmarks.
  arXiv  Detail & Related papers  (2024-09-05T03:22:22Z)
• MODRL-TA:A Multi-Objective Deep Reinforcement Learning Framework for Traffic Allocation in E-Commerce Search [13.893431289065997]
  This paper propose a multi-objective deep reinforcement learning framework consisting of multi-objective Q-learning (MOQ), a decision fusion algorithm (DFM) based on the cross-entropy method(CEM) and a progressive data augmentation system(PDA) Experiments on real-world online e-commerce systems demonstrate the significant improvements of MODRL-TA.
  arXiv  Detail & Related papers  (2024-07-22T08:40:27Z)
• Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation [52.930183136111864]
  We propose using scorable negotiation to evaluate Large Language Models (LLMs) To reach an agreement, agents must have strong arithmetic, inference, exploration, and planning capabilities. We provide procedures to create new games and increase games' difficulty to have an evolving benchmark.
  arXiv  Detail & Related papers  (2023-09-29T13:33:06Z)
• Dynamic Pricing with Volume Discounts in Online Settings [102.00782184214326]
  This paper focuses on pricing in e-commerce when objective function is profit and only transaction data are available. Our work aims to find a pricing strategy that allows defining optimal prices at different volume thresholds to serve different classes of users. We design a two-phase online learning algorithm, namely-B- capable of exploiting the data in an online fashion.
  arXiv  Detail & Related papers  (2022-11-17T16:01:06Z)
• Learning to Minimize Cost-to-Serve for Multi-Node Multi-Product Order Fulfilment in Electronic Commerce [3.3865605512957457]
  We find that the cost of delivery of products from the most node in the supply chain is a key challenge. The large scale, highproblemity, and large geographical spread of e-commerce supply chains make this setting ideal for a carefully designed data-driven decision-making algorithm. We show that a reinforcement learning based algorithm is competitive with these policies, with the potential of efficient scale-up in the real world.
  arXiv  Detail & Related papers  (2021-12-16T09:42:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.