Beyond the high score: Prosocial ability profiles of multi-agent populations

• URL: http://arxiv.org/abs/2509.14485v1
• Date: Wed, 17 Sep 2025 23:29:39 GMT
• Title: Beyond the high score: Prosocial ability profiles of multi-agent populations
• Authors: Marko Tesic, Yue Zhao, Joel Z. Leibo, Rakshit S. Trivedi, Jose Hernandez-Orallo,
• Abstract summary: The Melting Pot contest is a social AI evaluation suite designed to assess the cooperation capabilities of AI systems.<n>We apply a Bayesian approach known as Measurement Layouts to infer the capability profiles of multi-agent systems in the Melting Pot contest.<n>We show that these capability profiles not only predict future performance within the Melting Pot suite but also reveal the underlying prosocial abilities of agents.
• Score: 7.740015167057365
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The development and evaluation of social capabilities in AI agents require complex environments where competitive and cooperative behaviours naturally emerge. While game-theoretic properties can explain why certain teams or agent populations outperform others, more abstract behaviours, such as convention following, are harder to control in training and evaluation settings. The Melting Pot contest is a social AI evaluation suite designed to assess the cooperation capabilities of AI systems. In this paper, we apply a Bayesian approach known as Measurement Layouts to infer the capability profiles of multi-agent systems in the Melting Pot contest. We show that these capability profiles not only predict future performance within the Melting Pot suite but also reveal the underlying prosocial abilities of agents. Our analysis indicates that while higher prosocial capabilities sometimes correlate with better performance, this is not a universal trend-some lower-scoring agents exhibit stronger cooperation abilities. Furthermore, we find that top-performing contest submissions are more likely to achieve high scores in scenarios where prosocial capabilities are not required. These findings, together with reports that the contest winner used a hard-coded solution tailored to specific environments, suggest that at least one top-performing team may have optimised for conditions where cooperation was not necessary, potentially exploiting limitations in the evaluation framework. We provide recommendations for improving the annotation of cooperation demands and propose future research directions to account for biases introduced by different testing environments. Our results demonstrate that Measurement Layouts offer both strong predictive accuracy and actionable insights, contributing to a more transparent and generalisable approach to evaluating AI systems in complex social settings.

Related papers

• Evaluating Generalization Capabilities of LLM-Based Agents in Mixed-Motive Scenarios Using Concordia [100.74015791021044]
  Large Language Model (LLM) agents have demonstrated impressive capabilities for social interaction.<n>Existing evaluation methods fail to measure how well these capabilities generalize to novel social situations.<n>We present empirical results from the NeurIPS 2024 Concordia Contest, where agents were evaluated on their ability to achieve mutual gains.
  arXiv  Detail & Related papers  (2025-12-03T00:11:05Z)
• The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation [0.16921396880325779]
  We introduce a novel evaluation framework that uses multi-agent debate as a controlled "social laboratory"<n>We show that assigned personas induce stable, measurable psychometric profiles, particularly in cognitive effort.<n>This work provides a blueprint for a new class of dynamic, psychometrically grounded evaluation protocols.
  arXiv  Detail & Related papers  (2025-10-01T07:10:28Z)
• Towards Recommender Systems LLMs Playground (RecSysLLMsP): Exploring Polarization and Engagement in Simulated Social Networks [6.813586966214873]
  This paper introduces a novel simulation framework leveraging Large Language Models (LLMs) to explore the impacts of different content recommendation setups on user engagement and polarization in social networks.<n>By creating diverse AI agents with descriptive, static, and dynamic attributes, we assess their autonomous behaviour across three scenarios: Plurality, Balanced, and Similarity.<n>Our study emphasizes the need for a careful balance in recommender system designs to enhance user satisfaction while mitigating societal polarization.
  arXiv  Detail & Related papers  (2025-01-29T14:23:34Z)
• Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration [51.452664740963066]
  Collaborative Gym is a framework enabling asynchronous, tripartite interaction among agents, humans, and task environments.<n>We instantiate Co-Gym with three representative tasks in both simulated and real-world conditions.<n>Our findings reveal that collaborative agents consistently outperform their fully autonomous counterparts in task performance.
  arXiv  Detail & Related papers  (2024-12-20T09:21:15Z)
• Inverse Reinforcement Learning with Sub-optimal Experts [56.553106680769474]
  We study the theoretical properties of the class of reward functions that are compatible with a given set of experts. Our results show that the presence of multiple sub-optimal experts can significantly shrink the set of compatible rewards. We analyze a uniform sampling algorithm that results in being minimax optimal whenever the sub-optimal experts' performance level is sufficiently close to the one of the optimal agent.
  arXiv  Detail & Related papers  (2024-01-08T12:39:25Z)
• ProAgent: Building Proactive Cooperative Agents with Large Language Models [89.53040828210945]
  ProAgent is a novel framework that harnesses large language models to create proactive agents. ProAgent can analyze the present state, and infer the intentions of teammates from observations. ProAgent exhibits a high degree of modularity and interpretability, making it easily integrated into various coordination scenarios.
  arXiv  Detail & Related papers  (2023-08-22T10:36:56Z)
• Flexible social inference facilitates targeted social learning when rewards are not observable [58.762004496858836]
  Groups coordinate more effectively when individuals are able to learn from others' successes. We suggest that social inference capacities may help bridge this gap, allowing individuals to update their beliefs about others' underlying knowledge and success from observable trajectories of behavior.
  arXiv  Detail & Related papers  (2022-12-01T21:04:03Z)
• Melting Pot 2.0 [54.60680281014163]
  Melting Pot is a tool developed to facilitate work on multi-agent artificial intelligence. It provides an evaluation protocol that measures generalization to novel social partners. Melting Pot aims to cover a maximally diverse set of interdependencies and incentives.
  arXiv  Detail & Related papers  (2022-11-24T18:23:28Z)
• Warmth and competence in human-agent cooperation [0.7237068561453082]
  Recent studies demonstrate that AI agents trained with deep reinforcement learning are capable of collaborating with humans. We train deep reinforcement learning agents in Coins, a two-player social dilemma. Participants' perceptions of warmth and competence predict their stated preferences for different agents.
  arXiv  Detail & Related papers  (2022-01-31T18:57:08Z)
• On Emergent Communication in Competitive Multi-Agent Teams [116.95067289206919]
  We investigate whether competition for performance from an external, similar agent team could act as a social influence. Our results show that an external competitive influence leads to improved accuracy and generalization, as well as faster emergence of communicative languages.
  arXiv  Detail & Related papers  (2020-03-04T01:14:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.