Business Arena: Benchmarking LLM Agents in a Realistic Marketplace

• URL: http://arxiv.org/abs/2608.08621v1
• Date: Sun, 09 Aug 2026 10:08:21 GMT
• Title: Business Arena: Benchmarking LLM Agents in a Realistic Marketplace
• Abstract summary: Business Arena is a controlled environment where an AI agent runs a cross-border shop, buying from suppliers and selling to buyers over a long horizon.<n>We evaluate 15 frontier models and find a ninefold difference in mean final net worth.<n>Even the best model falls behind human-designed strategies, indicating that business operation remains challenging for LLM agents.
• Score: 14.641921772947926
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Running a business is a challenging form of intelligent work. Operators must infer opportunities from partial signals, commit capital under uncertainty, adapt to delayed outcomes in a changing market, and satisfy regulatory obligations before trading legally. Frontier LLM agents can increasingly complete complex workflows, yet business-related capabilities are rarely evaluated in existing agent benchmarks. We introduce \textbf{Business Arena}, a controlled environment where an AI agent runs a cross-border shop, buying from suppliers and selling to buyers over a long horizon. We ground the arena in real Alibaba.com sourcing data and market conditions calibrated from authoritative sources. Delayed and coupled consequences make individual business decisions difficult to judge, but their combined outcome is measurable through profit. Because profit alone cannot explain why an agent succeeds or fails, we compare agents with human-designed strategies to estimate available opportunity, use skill-level metrics to reveal underlying strengths and weaknesses, and trace realized gains and losses to the actions that produced them. We use mechanism ablations to establish that strong results reflect genuine business intelligence rather than neglect or simulator-specific shortcuts. We evaluate 15 frontier models and find a ninefold difference in mean final net worth. Even the best model falls behind human-designed strategies, indicating that business operation remains challenging for LLM agents. Skill-level analysis reveals operating styles, from margin-focused premium sellers to high-turnover wholesalers and customer-service specialists, while action-level attribution identifies the sourcing, pricing, and recovery decisions that create or destroy value. Together, Business Arena takes a first step toward a realistic and trustworthy testbed for evaluating end-to-end business agents.

Related papers

• Cattle Trade: A Multi-Agent Benchmark for LLM Bluffing, Bidding, and Bargaining [0.5691230599672109]
  We introduce textscCattle Trade, a benchmark for evaluating large language models (LLMs) as agents in strategic reasoning.<n>The benchmark combines auctions, hidden-offer trade challenges (TCs), bargaining, bluffing, opponent modeling, and resource allocation within a single long-horizon game.<n>We evaluate seven cost-efficient language models and three deterministic code agents across 242 games.
  arXiv  Detail & Related papers  (2026-05-14T08:20:03Z)
• PieArena: Frontier Language Agents Achieve MBA-Level Negotiation Performance and Reveal Novel Behavioral Differences [13.759960839511807]
  We introduce PieArena, a large-scale negotiation benchmark grounded in multi-agent interactions.<n>We find systematic evidence of human-expert-level performance in which a representative frontier language agent (GPT-5) matches or outperforms trained business-school students.
  arXiv  Detail & Related papers  (2026-02-05T04:52:20Z)
• TradeTrap: Are LLM-based Trading Agents Truly Reliable and Faithful? [44.01987401527335]
  TradeTrap is a unified evaluation framework for systematically stress-testing both adaptive and procedural autonomous trading agents.<n>It targets four core components of autonomous trading agents: market intelligence, strategy formulation, portfolio and ledger handling, and trade execution.<n>Experiments show that small perturbations at a single component can propagate through the agent decision loop and induce extreme concentration, runaway exposure, and large portfolio drawdowns.
  arXiv  Detail & Related papers  (2025-12-01T23:06:42Z)
• Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets [74.91125572848439]
  We study two-sided agentic marketplaces where Assistant agents represent consumers and Service agents represent competing businesses.<n>This environment enables us to study key market dynamics: the utility agents achieve, behavioral biases, vulnerability to manipulation, and how search mechanisms shape market outcomes.<n>Our experiments show that frontier models can approach optimal welfare-- but only under ideal search conditions. Performance degrades sharply with scale, and all models exhibit severe first-proposal bias, creating 10-30x advantages for response speed over quality.
  arXiv  Detail & Related papers  (2025-10-27T18:35:59Z)
• Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Sets [57.179679246370114]
  In financial applications, reinforcement learning (RL) agents are commonly trained on historical data, where their actions do not influence prices.<n>During deployment, these agents trade in live markets where their own transactions can shift asset prices, a phenomenon known as market impact.<n>Traditional robust RL approaches address this model misspecification by optimizing the worst-case performance over a set of uncertainties.<n>We develop a novel class of elliptic uncertainty sets, enabling efficient and tractable robust policy evaluation.
  arXiv  Detail & Related papers  (2025-10-22T18:22:25Z)
• When Agents Trade: Live Multi-Market Trading Benchmark for LLM Agents [74.55061622246824]
  Agent Market Arena (AMA) is the first lifelong, real-time benchmark for evaluating Large Language Model (LLM)-based trading agents.<n>AMA integrates verified trading data, expert-checked news, and diverse agent architectures within a unified trading framework.<n>It evaluates agents across GPT-4o, GPT-4.1, Claude-3.5-haiku, Claude-sonnet-4, and Gemini-2.0-flash.
  arXiv  Detail & Related papers  (2025-10-13T17:54:09Z)
• StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets? [44.10622904101254]
  Large language models (LLMs) have recently demonstrated strong capabilities as autonomous agents.<n>We introduce StockBench, a benchmark designed to evaluate LLM agents in realistic, multi-month stock trading environments.<n>Our evaluation shows that while most LLM agents struggle to outperform the simple buy-and-hold baseline, several models demonstrate the potential to deliver higher returns and manage risk more effectively.
  arXiv  Detail & Related papers  (2025-10-02T16:54:57Z)
• CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions [85.88573535033406]
  CRMArena-Pro is a novel benchmark for holistic, realistic assessment of LLM agents in diverse professional settings.<n>It incorporates multi-turn interactions guided by diverse personas and robust confidentiality awareness assessments.<n>Experiments reveal leading LLM agents achieve only around 58% single-turn success on CRMArena-Pro, with performance dropping significantly to approximately 35% in multi-turn settings.
  arXiv  Detail & Related papers  (2025-05-24T21:33:22Z)
• EmoDebt: Bayesian-Optimized Emotional Intelligence for Strategic Agent-to-Agent Debt Recovery [65.30120701878582]
  Large Language Model (LLM) agents are vulnerable to exploitation in emotion-sensitive domains like debt collection.<n>EmoDebt is an emotional intelligence engine that reframes a model's ability to express emotion in negotiation as a sequential decision-making problem.<n>EmoDebt achieves significant strategic robustness, substantially outperforming non-adaptive and emotion-agnostic baselines.
  arXiv  Detail & Related papers  (2025-03-27T01:41:34Z)
• When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environments [55.19252983108372]
  We have developed a multi-agent AI system called StockAgent, driven by LLMs. The StockAgent allows users to evaluate the impact of different external factors on investor trading. It avoids the test set leakage issue present in existing trading simulation systems based on AI Agents.
  arXiv  Detail & Related papers  (2024-07-15T06:49:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.