CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies

• URL: http://arxiv.org/abs/2606.16613v1
• Date: Mon, 15 Jun 2026 12:04:44 GMT
• Title: CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies
• Abstract summary: We introduce CoffeeBench, a benchmark for evaluating LLM agents in a long-horizon multi-agent economy composed of heterogeneous firms.<n>Two farmers, two roasters, and two retailers autonomously operate their businesses over a 90-day simulation.<n>All models outperform a passive baseline that takes no actions, with most achieving positive net income.
• Score: 2.545704705278791
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As LLM agents become capable of increasingly long-horizon tasks, evaluating their performance in economic systems is becoming increasingly important. Unlike existing benchmarks that primarily evaluate a single agent interacting with a passive environment, economic systems are inherently multi-agent, requiring autonomous agents to communicate, negotiate, and transact while pursuing their own objectives over extended periods. We introduce CoffeeBench, a benchmark for evaluating LLM agents in a long-horizon multi-agent economy composed of heterogeneous firms. In CoffeeBench, two farmers, two roasters, and two retailers autonomously operate their businesses over a 90-day simulation, each seeking to maximize cumulative net income through communication and transactions while managing cash, inventory, and pricing. The evaluated model controls one coffee roaster, while the remaining firms are controlled by fixed reference agents. Across several recent open-weight and proprietary LLMs, all models outperform a passive baseline that takes no actions, with most achieving positive net income. Analysis of agent behavior reveals substantial differences in long-horizon economic interaction: higher-performing models communicate more actively with other firms, whereas Claude~Haiku~4.5 exhibits an idle-drift failure mode, repeatedly choosing inaction despite producing coherent assessments and plans. We release our code and agent trajectories to support future research.

Related papers

• Multi$^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments [5.37133760455631]
  A central goal of large language model (LLM) research is to build agentic systems that can plan, act, and adapt through sustained interaction with dynamic environments.<n>We introduce Multi$2$, a hierarchical multi-agent decision-making framework that explicitly decomposes agent behavior into complementary roles.<n>A high-level agent (System 1) focuses on context-aware sub-goal generation using supervised fine-tuning (SFT), while a low-level agent (System 2) executes atomic actions through offline-to-online reinforcement learning (RL) in interactive environments.
  arXiv  Detail & Related papers  (2026-06-02T14:20:09Z)
• Cattle Trade: A Multi-Agent Benchmark for LLM Bluffing, Bidding, and Bargaining [0.5691230599672109]
  We introduce textscCattle Trade, a benchmark for evaluating large language models (LLMs) as agents in strategic reasoning.<n>The benchmark combines auctions, hidden-offer trade challenges (TCs), bargaining, bluffing, opponent modeling, and resource allocation within a single long-horizon game.<n>We evaluate seven cost-efficient language models and three deterministic code agents across 242 games.
  arXiv  Detail & Related papers  (2026-05-14T08:20:03Z)
• AgentCollab: A Self-Evaluation-Driven Collaboration Paradigm for Efficient LLM Agents [37.232397795331444]
  We present AgentCollab, a self-driven collaborative inference framework that dynamically coordinates models with different reasoning capacities during agent execution.<n>Instead of relying on external routing modules, the framework uses the agent's own self-reflection signal to determine whether the current reasoning trajectory is making meaningful progress.
  arXiv  Detail & Related papers  (2026-03-27T03:07:34Z)
• MARTI-MARS$^2$: Scaling Multi-Agent Self-Search via Reinforcement Learning for Code Generation [64.2621682259008]
  Multi-Agent Reinforced Training and Inference Framework with Self-Search Scaling (MARTI-MARS2)<n>We propose a Multi-Agent Reinforced Training and Inference Framework with Self-Search Scaling (MARTI-MARS2) to integrate policy learning with multi-agent tree search.<n>We show that MARTI-MARS2 achieves 77.7%, outperforming strong baselines like GPT-5.1 on challenging code generation benchmarks.
  arXiv  Detail & Related papers  (2026-02-08T07:28:44Z)
• AgenticPay: A Multi-Agent LLM Negotiation System for Buyer-Seller Transactions [49.49718899185783]
  Large language model (LLM)-based agents are increasingly expected to negotiate, coordinate, and transact autonomously.<n>We introduce AgenticPay, a benchmark and simulation framework for multi-agent buyer-seller negotiation driven by natural language.
  arXiv  Detail & Related papers  (2026-02-05T18:50:36Z)
• Collaborative Multi-Agent Test-Time Reinforcement Learning for Reasoning [112.16686518063456]
  We introduce textbfMulti-Agent Test-Time Reinforcement Learning (MATTRL), a framework that injects structured textual experience into multi-agent deliberation at inference time.<n>MATTRL forms a multi-expert team of specialists for multi-turn discussions, retrieves and integrates test-time experiences, and reaches consensus for final decision-making.<n>Across challenging benchmarks in medicine, math, and education, MATTRL improves accuracy by an average of 3.67% over a multi-agent baseline, and by 8.67% over comparable single-agent baselines
  arXiv  Detail & Related papers  (2026-01-14T17:57:43Z)
• When Agents Trade: Live Multi-Market Trading Benchmark for LLM Agents [74.55061622246824]
  Agent Market Arena (AMA) is the first lifelong, real-time benchmark for evaluating Large Language Model (LLM)-based trading agents.<n>AMA integrates verified trading data, expert-checked news, and diverse agent architectures within a unified trading framework.<n>It evaluates agents across GPT-4o, GPT-4.1, Claude-3.5-haiku, Claude-sonnet-4, and Gemini-2.0-flash.
  arXiv  Detail & Related papers  (2025-10-13T17:54:09Z)
• QuantAgents: Towards Multi-agent Financial System via Simulated Trading [40.636918662488505]
  QuantAgents is a multi-agent system integrating simulated trading.<n> QuantAgents comprises four agents: a simulated trading analyst, a risk control analyst, a market news analyst, and a manager.<n>Our system incentivizes agents to receive feedback on two fronts: performance in real-world markets and predictive accuracy in simulated trading.
  arXiv  Detail & Related papers  (2025-10-06T09:45:57Z)
• LMAgent: A Large-scale Multimodal Agents Society for Multi-user Simulation [66.52371505566815]
  Large language models (LLMs)-based AI agents have made significant progress, enabling them to achieve human-like intelligence.<n>We present LMAgent, a very large-scale and multimodal agents society based on multimodal LLMs.<n>In LMAgent, besides chatting with friends, the agents can autonomously browse, purchase, and review products, even perform live streaming e-commerce.
  arXiv  Detail & Related papers  (2024-12-12T12:47:09Z)
• AgentBench: Evaluating LLMs as Agents [99.12825098528212]
  Large Language Model (LLM) as agents has been widely acknowledged recently.<n>We present AgentBench, a benchmark that consists of 8 distinct environments to assess LLM-as-Agent's reasoning and decision-making abilities.
  arXiv  Detail & Related papers  (2023-08-07T16:08:11Z)
• Learning From Good Trajectories in Offline Multi-Agent Reinforcement Learning [98.07495732562654]
  offline multi-agent reinforcement learning (MARL) aims to learn effective multi-agent policies from pre-collected datasets. One agent learned by offline MARL often inherits this random policy, jeopardizing the performance of the entire team. We propose a novel framework called Shared Individual Trajectories (SIT) to address this problem.
  arXiv  Detail & Related papers  (2022-11-28T18:11:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.