Social Genome: Grounded Social Reasoning Abilities of Multimodal Models

• URL: http://arxiv.org/abs/2502.15109v2
• Date: Thu, 06 Mar 2025 11:07:48 GMT
• Title: Social Genome: Grounded Social Reasoning Abilities of Multimodal Models
• Authors: Leena Mathur, Marian Qian, Paul Pu Liang, Louis-Philippe Morency,
• Abstract summary: Social Genome is the first benchmark for fine-grained, grounded social reasoning abilities of multimodal models.<n>It contains 272 videos of interactions and 1,486 human-annotated reasoning traces related to inferences about these interactions.<n>Social Genome is also the first modeling challenge to study external knowledge in social reasoning.
• Score: 61.88413918026431
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Social reasoning abilities are crucial for AI systems to effectively interpret and respond to multimodal human communication and interaction within social contexts. We introduce Social Genome, the first benchmark for fine-grained, grounded social reasoning abilities of multimodal models. Social Genome contains 272 videos of interactions and 1,486 human-annotated reasoning traces related to inferences about these interactions. These traces contain 5,777 reasoning steps that reference evidence from visual cues, verbal cues, vocal cues, and external knowledge (contextual knowledge external to videos). Social Genome is also the first modeling challenge to study external knowledge in social reasoning. Social Genome computes metrics to holistically evaluate semantic and structural qualities of model-generated social reasoning traces. We demonstrate the utility of Social Genome through experiments with state-of-the-art models, identifying performance gaps and opportunities for future research to improve the grounded social reasoning abilities of multimodal models.

Related papers

• The Human Robot Social Interaction (HSRI) Dataset: Benchmarking Foundational Models' Social Reasoning [49.32390524168273]
  Our work aims to advance the social reasoning of embodied artificial intelligence (AI) agents in real-world social interactions. We introduce a large-scale real-world Human Robot Social Interaction (HSRI) dataset to benchmark the capabilities of language models (LMs) and foundational models (FMs) Our dataset consists of 400 real-world human social robot interaction videos and over 10K annotations, detailing the robot's social errors, competencies, rationale, and corrective actions.
  arXiv  Detail & Related papers  (2025-04-07T06:27:02Z)
• Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questions [67.60397632819202]
  Building socially-intelligent AI agents (Social-AI) is a multidisciplinary, multimodal research goal. We identify a set of underlying technical challenges and open questions for researchers across computing communities to advance Social-AI.
  arXiv  Detail & Related papers  (2024-04-17T02:57:42Z)
• SocialDial: A Benchmark for Socially-Aware Dialogue Systems [45.3266270265532]
  We present the first socially-aware dialogue corpus - SocialDial, based on Chinese social culture. SocialDial consists of two parts: 1,563 multi-turn dialogues between two human speakers with fine-grained labels, and 4,870 synthetic conversations generated by ChatGPT. The human corpus covers five categories of social norms, which have 14 sub-categories in total.
  arXiv  Detail & Related papers  (2023-04-24T11:55:22Z)
• Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs [77.88043871260466]
  We show that one of today's largest language models lacks this kind of social intelligence out-of-the box. We conclude that person-centric NLP approaches might be more effective towards neural Theory of Mind.
  arXiv  Detail & Related papers  (2022-10-24T14:58:58Z)
• Predicting Opinion Dynamics via Sociologically-Informed Neural Networks [31.77040611129394]
  We present Sociologically-Informed Neural Network (SINN), which integrates theoretical models and social media data. In particular, we recast theoretical models as ordinary differential equations (ODEs) We train a neural network that simultaneously approximates the data and conforms to the ODEs that represent the social scientific knowledge.
  arXiv  Detail & Related papers  (2022-07-07T05:55:47Z)
• Social Neuro AI: Social Interaction as the "dark matter" of AI [0.0]
  We argue that empirical results from social psychology and social neuroscience along with the framework of dynamics can be of inspiration to the development of more intelligent artificial agents.
  arXiv  Detail & Related papers  (2021-12-31T13:41:53Z)
• PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception [50.551003004553806]
  We create a dataset of physically-grounded abstract social events, PHASE, that resemble a wide range of real-life social interactions. Phase is validated with human experiments demonstrating that humans perceive rich interactions in the social events. As a baseline model, we introduce a Bayesian inverse planning approach, SIMPLE, which outperforms state-of-the-art feed-forward neural networks.
  arXiv  Detail & Related papers  (2021-03-02T18:44:57Z)
• Graph-Based Social Relation Reasoning [101.9402771161935]
  We propose a graph relational reasoning network (GR2N) for social relation recognition. Our method considers the paradigm of jointly inferring the relations by constructing a social relation graph. Experimental results illustrate that our method generates a reasonable and consistent social relation graph.
  arXiv  Detail & Related papers  (2020-07-15T03:01:11Z)
• Theory In, Theory Out: The uses of social theory in machine learning for social science [3.180013942295509]
  We show how social theory can be used to answer the basic methodological and interpretive questions that arise at each stage of the machine learning pipeline. We believe this paper can act as a guide for computer and social scientists alike to navigate the substantive questions involved in applying the tools of machine learning to social data.
  arXiv  Detail & Related papers  (2020-01-09T20:04:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.