RetailBench: Benchmarking long horizon reasoning and coherent decision making of LLM agents in realistic retail environments

• URL: http://arxiv.org/abs/2606.15862v3
• Date: Fri, 19 Jun 2026 10:48:19 GMT
• Title: RetailBench: Benchmarking long horizon reasoning and coherent decision making of LLM agents in realistic retail environments
• Abstract summary: Large language model (LLM) agents have made rapid progress on short-horizon, well-scoped tasks, yet their ability to sustain coherent decisions in dynamic long-horizon environments remains uncertain.<n>We introduce RetailBench, a data-grounded simulation benchmark for evaluating tool-using LLM agents in single-store supermarket operation.
• Score: 8.751899157366005
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Large language model (LLM) agents have made rapid progress on short-horizon, well-scoped tasks, yet their ability to sustain coherent decisions in dynamic long-horizon environments remains uncertain. We introduce RetailBench, a data-grounded simulation benchmark for evaluating tool-using LLM agents in single-store supermarket operation. RetailBench models retail management as a partially observable decision process and is designed to support thousand-day-scale simulations. In this environment, agents must manage pricing, replenishment, supplier selection, shelf assortment, inventory aging, customer feedback, external events, and cash-flow constraints. We evaluate seven contemporary LLMs under representative agent frameworks over a 180-day evaluation horizon and compare them with a privileged oracle policy. Results show substantial variation across models: only a small subset survives the full evaluation horizon, and even the strongest LLM runs remain substantially behind the oracle policy in final net worth and sales outcomes. Behavioral analysis attributes these gaps to incomplete evidence acquisition, surface-level decision making, and the lack of a consistent long-horizon policy. RetailBench provides a controlled testbed for studying reliable autonomy in economically grounded long-horizon decision-making.

Related papers

• MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations [18.706607921548326]
  Long-Term Coherence is the capacity to preserve purposeful behavior across extended horizons while adapting decisions to accumulated evidence.<n>Seller-side e-commerce provides a suitable setting for this evaluation through recurrent and interdependent decisions.<n> MerchantBench is a 365-day order-level simulation grounded in 98,843 real e-commerce product records and equipped with 26 tools for agent interaction.
  arXiv  Detail & Related papers  (2026-07-31T02:20:24Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• Market-Bench: Benchmarking Large Language Models on Economic and Trade Competition [102.73682784993169]
  This paper introduces textbfMarket-Bench, a benchmark that evaluates the capabilities of large language models (LLMs) in economically-relevant tasks.<n>We construct a multi-agent supply chain economic model where LLMs act as retailer agents responsible for procuring and retailing merchandise.<n>Market-Bench logs complete trajectories of bids, prices, slogans, sales, and balance-sheet states, enabling automatic evaluation with economic, operational, and semantic metrics.
  arXiv  Detail & Related papers  (2026-04-07T07:23:51Z)
• Can LLM Agents Be CFOs? A Benchmark for Resource Allocation in Dynamic Enterprise Environments [39.051947374027435]
  We introduce EnterpriseArena, the first benchmark for evaluating agents on long-horizon enterprise resource allocation.<n>It instantiates CFO-style decision-making in a 132-month enterprise simulator combining firm-level financial data, anonymized business documents, macroeconomic and industry signals, and expert-validated operating rules.<n>Experiments show that only 16% of runs survive the full horizon, and larger models do not reliably outperform smaller ones.
  arXiv  Detail & Related papers  (2026-03-24T18:25:00Z)
• RetailBench: Evaluating Long-Horizon Autonomous Decision-Making and Strategy Stability of LLM Agents in Realistic Retail Environments [8.751899157366005]
  Large Language Model (LLM)-based agents have achieved notable success on short-horizon and highly structured tasks.<n>We introduce RetailBench, a high-fidelity benchmark designed to evaluate long-horizon autonomous decision-making in realistic commercial scenarios.<n>We propose the Evolving Strategy & Execution framework, which separates high-level strategic reasoning from low-level action execution.
  arXiv  Detail & Related papers  (2026-03-17T12:35:52Z)
• EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economies [61.267836443398124]
  We introduce EcoGym, a benchmark for continuous plan-and-execute decision making in interactive economies.<n>EcoGym is released as an open, testbed for transparent long-horizonity agent evaluation and for studying controllability-utility trade-offs in realistic economic settings.
  arXiv  Detail & Related papers  (2026-02-10T08:12:23Z)
• AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress [71.02263260394261]
  Large language models (LLMs) still encounter challenges in multi-turn decision-making tasks.<n>We build process reward models (PRMs) to evaluate each decision and guide the agent's decision-making process.<n>AgentPRM captures both the interdependence between sequential decisions and their contribution to the final goal.
  arXiv  Detail & Related papers  (2025-11-11T14:57:54Z)
• StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets? [44.10622904101254]
  Large language models (LLMs) have recently demonstrated strong capabilities as autonomous agents.<n>We introduce StockBench, a benchmark designed to evaluate LLM agents in realistic, multi-month stock trading environments.<n>Our evaluation shows that while most LLM agents struggle to outperform the simple buy-and-hold baseline, several models demonstrate the potential to deliver higher returns and manage risk more effectively.
  arXiv  Detail & Related papers  (2025-10-02T16:54:57Z)
• AI Playing Business Games: Benchmarking Large Language Models on Managerial Decision-Making in Dynamic Simulations [0.0]
  This research analyses a novel benchmark using a business game for the decision making in business.<n>The research contributes to the recent literature on AI by proposing a reproducible, open-access management simulator.
  arXiv  Detail & Related papers  (2025-09-30T14:43:05Z)
• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents [74.16170899755281]
  We introduce AgentBoard, a pioneering comprehensive benchmark and accompanied open-source evaluation framework tailored to analytical evaluation of LLM agents.<n>AgentBoard offers a fine-grained progress rate metric that captures incremental advancements as well as a comprehensive evaluation toolkit.<n>This not only sheds light on the capabilities and limitations of LLM agents but also propels the interpretability of their performance to the forefront.
  arXiv  Detail & Related papers  (2024-01-24T01:51:00Z)
• AgentBench: Evaluating LLMs as Agents [99.12825098528212]
  Large Language Model (LLM) as agents has been widely acknowledged recently.<n>We present AgentBench, a benchmark that consists of 8 distinct environments to assess LLM-as-Agent's reasoning and decision-making abilities.
  arXiv  Detail & Related papers  (2023-08-07T16:08:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.