Communicate-Predict-Act: Evaluating Social Intelligence of Agents

• URL: http://arxiv.org/abs/2604.08727v1
• Date: Thu, 09 Apr 2026 19:40:15 GMT
• Title: Communicate-Predict-Act: Evaluating Social Intelligence of Agents
• Abstract summary: We introduce a multiplayer arena of mixed cooperative and competitive social games to study large language model (LLM) social intelligence.<n>We analyze gameplay traces to extract sociocognitive metrics capturing action prediction, communicative influence, strategic reasoning, and tradeoffs under conflicting interests.
• Score: 15.575081867336529
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: As large language model (LLM) agents become more prevalent in real world social settings, social intelligence will play an increasingly critical role. But social intelligence is still a poorly defined construct, for humans and artificial agents. We introduce a multiplayer arena of mixed cooperative and competitive social games to study LLM social intelligence. The controllability of LLM based agents enables systematic evaluation, which also supports broader inferences about social intelligence per se. We evaluated eight diverse LLMs (24B to 1T parameters) using a Communicate Predict Act (COMPACT) interaction protocol and fine grained probing of social dynamics. Elo style ratings reveal consistent performance differences across models, but this scalar measure provides only a partial characterization of social intelligence. To address this limitation, we analyze gameplay traces to extract sociocognitive metrics capturing action prediction, communicative influence, strategic reasoning, and tradeoffs under conflicting interests. These sociocognitive metrics exhibit strong intramodel consistency and they reliably predict pairwise agent advantage in game outcomes (AUC ROC = 0.82). Feature importance analysis indicates that surprisingly, influence, transparency, and adaptability are more predictive of success than Theory of Mind inference or deep planning. Together, our results advance a testable, multidimensional conception of social intelligence and provide empirical insights into the capacities that underpin it.

Related papers

• One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligence [25.89075578734277]
  This paper introduces OMAR: One Model, All Roles, a reinforcement learning framework for AI.<n>OMAR allows a single model to role-play all participants in a conversation simultaneously, learning to achieve long-term goals and complex social norms.<n>We show that trained models develop fine-grained, emergent social intelligence, such as empathy, persuasion, and compromise seeking.
  arXiv  Detail & Related papers  (2026-02-03T05:09:49Z)
• SocialEval: Evaluating Social Intelligence of Large Language Models [70.90981021629021]
  Social Intelligence (SI) equips humans with interpersonal abilities to behave wisely in navigating social interactions to achieve social goals.<n>This presents an operational evaluation paradigm: outcome-oriented goal achievement evaluation and process-oriented interpersonal ability evaluation.<n>We propose SocialEval, a script-based bilingual SI benchmark, integrating outcome- and process-oriented evaluation by manually crafting narrative scripts.
  arXiv  Detail & Related papers  (2025-06-01T08:36:51Z)
• EgoSocialArena: Benchmarking the Social Intelligence of Large Language Models from a First-person Perspective [22.30892836263764]
  Social intelligence is built upon three pillars: cognitive intelligence, situational intelligence, and behavioral intelligence.<n>EgoSocialArena aims to systematically evaluate the social intelligence of large language models from a first-person perspective.
  arXiv  Detail & Related papers  (2024-10-08T16:55:51Z)
• InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Context [27.740204336800687]
  Large language models (LLMs) have demonstrated the potential to mimic human social intelligence. We develop a novel framework, InterIntent, to assess LLMs' social intelligence by mapping their ability to understand and manage intentions in a game setting.
  arXiv  Detail & Related papers  (2024-06-18T02:02:15Z)
• SocialBench: Sociality Evaluation of Role-Playing Conversational Agents [85.6641890712617]
  Large language models (LLMs) have advanced the development of various AI conversational agents. SocialBench is the first benchmark designed to evaluate the sociality of role-playing conversational agents at both individual and group levels. We find that agents excelling in individual level does not imply their proficiency in group level.
  arXiv  Detail & Related papers  (2024-03-20T15:38:36Z)
• SOTOPIA-$π$: Interactive Learning of Socially Intelligent Language Agents [73.35393511272791]
  We propose an interactive learning method, SOTOPIA-$pi$, improving the social intelligence of language agents. This method leverages behavior cloning and self-reinforcement training on filtered social interaction data according to large language model (LLM) ratings.
  arXiv  Detail & Related papers  (2024-03-13T17:17:48Z)
• Academically intelligent LLMs are not necessarily socially intelligent [56.452845189961444]
  The academic intelligence of large language models (LLMs) has made remarkable progress in recent times, but their social intelligence performance remains unclear. Inspired by established human social intelligence frameworks, we have developed a standardized social intelligence test based on real-world social scenarios.
  arXiv  Detail & Related papers  (2024-03-11T10:35:53Z)
• DeSIQ: Towards an Unbiased, Challenging Benchmark for Social Intelligence Understanding [60.84356161106069]
  We study the soundness of Social-IQ, a dataset of multiple-choice questions on videos of complex social interactions. Our analysis reveals that Social-IQ contains substantial biases, which can be exploited by a moderately strong language model. We introduce DeSIQ, a new challenging dataset, constructed by applying simple perturbations to Social-IQ.
  arXiv  Detail & Related papers  (2023-10-24T06:21:34Z)
• SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents [107.4138224020773]
  We present SOTOPIA, an open-ended environment to simulate complex social interactions between artificial agents and humans. In our environment, agents role-play and interact under a wide variety of scenarios; they coordinate, collaborate, exchange, and compete with each other to achieve complex social goals. We find that GPT-4 achieves a significantly lower goal completion rate than humans and struggles to exhibit social commonsense reasoning and strategic communication skills.
  arXiv  Detail & Related papers  (2023-10-18T02:27:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.