SMAC-Talk: A Natural Language Extension of the StarCraft Multi-Agent Challenge for Large Language Models

• URL: http://arxiv.org/abs/2606.04202v1
• Date: Tue, 02 Jun 2026 20:40:04 GMT
• Title: SMAC-Talk: A Natural Language Extension of the StarCraft Multi-Agent Challenge for Large Language Models
• Abstract summary: We introduce SMAC-Talk, a natural language extension of the StarCraft Multi-Agent Challenge for evaluating LLM-based agents in cooperative multi-agent environments.<n>SMAC-Talk includes a natural language communication channel which is used to probe agent coordination and trust.<n>We use this communication channel to construct different evaluation scenarios, including settings with an embedded communicator that tries to disrupt and deceive allies through communication alone.
• Score: 5.960036206015683
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: As LLMs become more widely deployed, they are increasingly expected to work alongside other AI agents rather than operating in isolation. Effective coordination in these settings requires agents to communicate, share information and make decisions under uncertainty. We introduce SMAC-Talk, a natural language extension of the StarCraft Multi-Agent Challenge for evaluating LLM-based agents in cooperative multi-agent environments. The environment has several key features such as decentralized control, partial observability and long-horizon decision making. SMAC-Talk includes a natural language communication channel which is used to probe agent coordination and trust. We use this communication channel to construct different evaluation scenarios, including settings with an embedded deceptive communicator that tries to disrupt and deceive allies through communication alone. We provide three agents for benchmarking using 4 models from the Qwen3.5 family and study how reasoning structure, memory and model scale affect coordination between agents. We release SMAC-Talk as an open benchmark to support the research community in developing and evaluating LLM agents in cooperative multi-agent settings.

Related papers

• Communication-Efficient Digital-Twin Coordination for Heterogeneous LLM Embodied Agents over Computing Power Networks [48.90302419915548]
  Embodied agent teams powered by heterogeneous large language models (LLMs) are being widely deployed in smart factories, warehouses, and service robotics.<n>We propose LDT-Coord, a networked coordination framework built upon a lightweight digital twin (DT)<n>Specifically, each agent independently selects its intended action and reports both the action decision and a structured temporal constraint over shared resources to the DT server.
  arXiv  Detail & Related papers  (2026-07-10T12:10:31Z)
• Benchmarking Open-Ended Multi-Agent Coordination in Language Agents [48.1363632826625]
  Alem embeds procedurally generated coordination tasks, soft specialisation, communication, and controllable coordination difficulty into a long-horizon survival world.<n>We introduce $alem$, a JAX-based benchmark for open-ended multi-agent coordination built on Craftax-like dynamics.
  arXiv  Detail & Related papers  (2026-06-06T21:13:43Z)
• When Numbers Start Talking: Implicit Numerical Coordination Among LLM-Based Agents [45.43445469098021]
  This paper presents a game-theoretic study of covert communication in multi-agent systems.<n>We analyse interactions across four canonical game-theoretic settings under different communication regimes.<n>Considering heterogeneous agent personalities and both one-shot and repeated games, we characterise when covert signals emerge and how they shape coordination and strategic outcomes.
  arXiv  Detail & Related papers  (2026-01-07T12:07:48Z)
• Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advances [61.539442227802226]
  Large Language Models (LLMs) have been integrated into Autonomous Driving Systems (ADSs) to support high-level decision-making.<n>LLMs face three major challenges: limited perception, insufficient collaboration, and high computational demands.<n>Recent advances in multi-agent ADSs leverage language-driven communication and coordination to enhance inter-agent collaboration.
  arXiv  Detail & Related papers  (2025-02-24T03:26:13Z)
• COMMA: A Communicative Multimodal Multi-Agent Benchmark [15.329501174451677]
  We introduce COMMA: a novel puzzle benchmark designed to evaluate the collaborative performance of multimodal multi-agent systems.<n>Our findings reveal surprising weaknesses in state-of-the-art models, including strong proprietary models like GPT-4o and reasoning models like o4-mini.<n>Many chain of thought reasoning models such as R1-Onevision and LLaVA-CoT struggle to outperform even a random baseline in agent-agent collaboration.
  arXiv  Detail & Related papers  (2024-10-10T02:49:47Z)
• SpeechAgents: Human-Communication Simulation with Multi-Modal Multi-Agent Systems [53.94772445896213]
  Large Language Model (LLM)-based multi-agent systems have demonstrated promising performance in simulating human society. We propose SpeechAgents, a multi-modal LLM based multi-agent system designed for simulating human communication.
  arXiv  Detail & Related papers  (2024-01-08T15:01:08Z)
• Large Language Model Enhanced Multi-Agent Systems for 6G Communications [94.45712802626794]
  We propose a multi-agent system with customized communication knowledge and tools for solving communication related tasks using natural language. We validate the effectiveness of the proposed multi-agent system by designing a semantic communication system.
  arXiv  Detail & Related papers  (2023-12-13T02:35:57Z)
• Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation [52.930183136111864]
  We propose using scorable negotiation to evaluate Large Language Models (LLMs) To reach an agreement, agents must have strong arithmetic, inference, exploration, and planning capabilities. We provide procedures to create new games and increase games' difficulty to have an evolving benchmark.
  arXiv  Detail & Related papers  (2023-09-29T13:33:06Z)
• Building Cooperative Embodied Agents Modularly with Large Language Models [104.57849816689559]
  We address challenging multi-agent cooperation problems with decentralized control, raw sensory observations, costly communication, and multi-objective tasks instantiated in various embodied environments. We harness the commonsense knowledge, reasoning ability, language comprehension, and text generation prowess of LLMs and seamlessly incorporate them into a cognitive-inspired modular framework. Our experiments on C-WAH and TDW-MAT demonstrate that CoELA driven by GPT-4 can surpass strong planning-based methods and exhibit emergent effective communication.
  arXiv  Detail & Related papers  (2023-07-05T17:59:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.