SocialGrid: A Benchmark for Planning and Social Reasoning in Embodied Multi-Agent Systems

• URL: http://arxiv.org/abs/2604.16022v1
• Date: Fri, 17 Apr 2026 12:51:46 GMT
• Title: SocialGrid: A Benchmark for Planning and Social Reasoning in Embodied Multi-Agent Systems
• Abstract summary: We introduce SocialGrid, an embodied multi-agent environment that evaluates agents on planning, task execution, and social reasoning.<n>Our evaluations reveal that even the strongest open model achieves below 60% accuracy in task completion and planning.<n>SocialGrid provides automatic failure analysis and fine-grained metrics, enabling developers to diagnose and improve their agents.
• Score: 38.033245595823026
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As Large Language Models (LLMs) transition from text processors to autonomous agents, evaluating their social reasoning in embodied multi-agent settings becomes critical. We introduce SocialGrid, an embodied multi-agent environment inspired by Among Us that evaluates LLM agents on planning, task execution, and social reasoning. Our evaluations reveal that even the strongest open model (GPT-OSS-120B) achieves below 60% accuracy in task completion and planning, with agents getting stuck in repetitive behaviors or failing to navigate basic obstacles. Since poor navigation confounds evaluation of social intelligence, SocialGrid offers an optional Planning Oracle to isolate social reasoning from planning deficits. While planning assistance improves task completion, social reasoning remains a bottleneck: agents fail to detect deception at near-random chance regardless of scale, relying on shallow heuristics rather than accumulating behavioral evidence. SocialGrid provides automatic failure analysis and fine-grained metrics, enabling developers to diagnose and improve their agents. We also establish a competitive leaderboard using Elo ratings from adversarial league play.

Related papers

• SocialRL: Refining LLMs' Social Intelligence through Multi-turn Reinforcement Learning and Reward Design [52.69818441707104]
  Social intelligence enables agents to read social context, infer intent, and adapt over sustained dialogue.<n>Existing reinforcement learning methods optimize single-turn utterances and sparse outcome rewards.<n>We propose SocialRL, a multi-turn reinforcement learning framework addressing both challenges.
  arXiv  Detail & Related papers  (2026-09-09T06:06:34Z)
• SocialBuddy: Tailoring Search Agent for Social Scenarios [7.562006814424155]
  SocialBuddy is the first agentic search framework tailored for social scenarios.<n>SocialEnv is the first large-scale simulated environment for social search.<n>SocialPO is a hybrid-granularity optimization framework.
  arXiv  Detail & Related papers  (2026-08-28T07:34:31Z)
• Interpretable Debiasing of Vision-Language Models for Social Fairness [55.85977929985967]
  We introduce an interpretable, model-agnostic bias mitigation framework, DeBiasLens, that localizes social attribute neurons in Vision-Language models.<n>We train SAEs on facial image or caption datasets without corresponding social attribute labels to uncover neurons highly responsive to specific demographics.<n>Our research lays the groundwork for future auditing tools, prioritizing social fairness in emerging real-world AI systems.
  arXiv  Detail & Related papers  (2026-02-27T13:37:11Z)
• From Obstacles to Etiquette: Robot Social Navigation with VLM-Informed Path Selection [57.74400052368147]
  This paper presents a social robot navigation framework that integrates geometric planning with contextual social reasoning.<n>The system first extracts obstacles and human dynamics to generate geometrically feasible candidate paths, then leverages a fine-tuned vision-language model (VLM) to evaluate these paths.<n>Experiments in four social navigation contexts demonstrate that our method achieves the best overall performance with the lowest personal space violation duration, the minimal pedestrian-facing time, and no social zone intrusions.
  arXiv  Detail & Related papers  (2026-02-09T18:46:12Z)
• GSM-Agent: Understanding Agentic Reasoning Using Controllable Environments [56.007498767771075]
  GSM-Agent is a novel benchmark for evaluating agentic reasoning in complex environments.<n>We analyze the agentic reasoning patterns by cluster the environment's document embeddings into nodes, and map each tool call to its nearest node.<n>We propose a tool-augmented test-time scaling method to improve LLM's agentic reasoning performance by adding tools to encourage models to revisit.
  arXiv  Detail & Related papers  (2025-09-26T07:24:37Z)
• SocialMaze: A Benchmark for Evaluating Social Reasoning in Large Language Models [41.68365456601248]
  We introduce SocialMaze, a new benchmark specifically designed to evaluate social reasoning.<n>SocialMaze systematically incorporates three core challenges: deep reasoning, dynamic interaction, and information uncertainty.<n>It provides six diverse tasks across three key settings: social reasoning games, daily-life interactions, and digital community platforms.
  arXiv  Detail & Related papers  (2025-05-29T17:47:36Z)
• SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially? [39.436399936544795]
  We present Strategic Planning, Interaction, and Negotiation (SPIN-Bench)<n>SPIN-Bench is a new multi-domain evaluation designed to measure the intelligence of strategic planning and social reasoning.
  arXiv  Detail & Related papers  (2025-03-16T04:10:53Z)
• AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios [38.878966229688054]
  We introduce AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios. Drawing on Dramaturgical Theory, AgentSense employs a bottom-up approach to create 1,225 diverse social scenarios constructed from extensive scripts. We analyze goals using ERG theory and conduct comprehensive experiments. Our findings highlight that LLMs struggle with goals in complex social scenarios, especially high-level growth needs, and even GPT-4o requires improvement in private information reasoning.
  arXiv  Detail & Related papers  (2024-10-25T07:04:16Z)
• Towards Socially Intelligent Agents with Mental State Transition and Human Utility [97.01430011496576]
  We propose to incorporate a mental state and utility model into dialogue agents. The hybrid mental state extracts information from both the dialogue and event observations. The utility model is a ranking model that learns human preferences from a crowd-sourced social commonsense dataset.
  arXiv  Detail & Related papers  (2021-03-12T00:06:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.