MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations

• URL: http://arxiv.org/abs/2607.28956v2
• Date: Tue, 04 Aug 2026 11:17:07 GMT
• Title: MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations
• Abstract summary: Long-Term Coherence is the capacity to preserve purposeful behavior across extended horizons while adapting decisions to accumulated evidence.<n>Seller-side e-commerce provides a suitable setting for this evaluation through recurrent and interdependent decisions.<n> MerchantBench is a 365-day order-level simulation grounded in 98,843 real e-commerce product records and equipped with 26 tools for agent interaction.
• Score: 18.706607921548326
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language model agents are increasingly evaluated as autonomous tool users, yet most benchmarks focus on bounded tasks with immediate success criteria. Real-world deployments often require Long-Term Coherence, the capacity to preserve purposeful behavior across extended horizons while adapting decisions to accumulated evidence. Evaluating this capacity requires a persistent environment in which actions constrain future choices, feedback arrives at heterogeneous delays, and incoherent behavior produces measurable cumulative effects. Seller-side e-commerce provides a suitable setting for this evaluation through recurrent and interdependent decisions over Product Sourcing, Listing and Pricing Control, Cash-Flow Management, and Mixed-Latency Feedback Adaptation. We introduce MerchantBench, a 365-day order-level simulation grounded in 98,843 real e-commerce product records and equipped with 26 tools for agent interaction. MerchantBench couples promptly observable Upstream Supplier Events with delayed Downstream Order Outcomes, requiring agents to follow individual order lifecycles and revisit earlier decisions. We evaluate eight LLMs under two agent frameworks in 48 runs, each spanning 365 simulated days. Our results reveal a substantial gap between even the latest LLMs and human participants, with the best LLM configuration attaining only 27.3\% of the mean final net assets achieved by human participants.

Related papers

• E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation [54.50499240166176]
  We introduce E-Commerce Bench, the first open-source benchmark that integrates multi-round counterpart negotiation and dynamic events into a year-long business operation.<n>We evaluate 18 frontier models across seven dimensions, including year-end assets, and find that no single model dominates.
  arXiv  Detail & Related papers  (2026-08-31T13:03:57Z)
• DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows [69.12339622053588]
  We introduce DuMateBench, a real-session benchmark reconstructed from anonymized and privacy-screened user sessions.<n>The resulting benchmark comprises 200 tasks spanning 8 broad scenarios and 17 fine-grained capability categories.<n>We execute these tasks in isolated Docker containers injected with three forms of real-world environmental complexity: Insufficient, Unstable, and Noisy.
  arXiv  Detail & Related papers  (2026-08-27T02:36:23Z)
• LLM-as-a-Verifier: A General-Purpose Verification Framework [74.40111651545979]
  We introduce LLM-as-a-Verifier, a general-purpose verification framework.<n>It provides fine-grained feedback for agentic tasks without requiring additional training.<n>It achieves state-of-the-art performance on several benchmarks.
  arXiv  Detail & Related papers  (2026-07-06T17:59:35Z)
• CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies [2.545704705278791]
  We introduce CoffeeBench, a benchmark for evaluating LLM agents in a long-horizon multi-agent economy composed of heterogeneous firms.<n>Two farmers, two roasters, and two retailers autonomously operate their businesses over a 90-day simulation.<n>All models outperform a passive baseline that takes no actions, with most achieving positive net income.
  arXiv  Detail & Related papers  (2026-06-15T12:04:44Z)
• RetailBench: Benchmarking long horizon reasoning and coherent decision making of LLM agents in realistic retail environments [8.751899157366005]
  Large language model (LLM) agents have made rapid progress on short-horizon, well-scoped tasks, yet their ability to sustain coherent decisions in dynamic long-horizon environments remains uncertain.<n>We introduce RetailBench, a data-grounded simulation benchmark for evaluating tool-using LLM agents in single-store supermarket operation.
  arXiv  Detail & Related papers  (2026-06-14T15:30:30Z)
• ShopSimulator: Evaluating and Exploring RL-Driven LLM Agent for Shopping Assistants [30.950450014358722]
  We introduce ShopSimulator, a large-scale and challenging Chinese shopping environment.<n>We evaluate LLMs across diverse scenarios, finding that even the best-performing models achieve less than 40% full-success rate.<n>Further training exploration provides practical guidance for overcoming these weaknesses.
  arXiv  Detail & Related papers  (2026-01-26T07:24:28Z)
• EComStage: Stage-wise and Orientation-specific Benchmarking for Large Language Models in E-commerce [26.028479108472265]
  Large Language Model (LLM)-based agents are increasingly deployed in e-commerce applications.<n>EComStage is a unified benchmark for evaluating agent-capable LLMs across the comprehensive stage-wise reasoning process.<n>We evaluate over 30 LLMs, spanning from 1B to over 200B parameters, including open-source models and closed-source APIs.
  arXiv  Detail & Related papers  (2026-01-06T06:39:16Z)
• AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress [71.02263260394261]
  Large language models (LLMs) still encounter challenges in multi-turn decision-making tasks.<n>We build process reward models (PRMs) to evaluate each decision and guide the agent's decision-making process.<n>AgentPRM captures both the interdependence between sequential decisions and their contribution to the final goal.
  arXiv  Detail & Related papers  (2025-11-11T14:57:54Z)
• AgentBench: Evaluating LLMs as Agents [99.12825098528212]
  Large Language Model (LLM) as agents has been widely acknowledged recently.<n>We present AgentBench, a benchmark that consists of 8 distinct environments to assess LLM-as-Agent's reasoning and decision-making abilities.
  arXiv  Detail & Related papers  (2023-08-07T16:08:11Z)
• Learning Multi-Agent Intention-Aware Communication for Optimal Multi-Order Execution in Finance [96.73189436721465]
  We first present a multi-agent RL (MARL) method for multi-order execution considering practical constraints. We propose a learnable multi-round communication protocol, for the agents communicating the intended actions with each other. Experiments on the data from two real-world markets have illustrated superior performance with significantly better collaboration effectiveness.
  arXiv  Detail & Related papers  (2023-07-06T16:45:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.