SocialGen: Modeling Multi-Human Social Interaction with Language Models

• URL: http://arxiv.org/abs/2503.22906v1
• Date: Fri, 28 Mar 2025 22:57:25 GMT
• Title: SocialGen: Modeling Multi-Human Social Interaction with Language Models
• Authors: Heng Yu, Juze Zhang, Changan Chen, Tiange Xiang, Yusu Fang, Juan Carlos Niebles, Ehsan Adeli,
• Abstract summary: We introduce SocialGen, the first unified motion-language model capable of modeling interaction behaviors among varying numbers of individuals.<n>Unlike prior methods that are limited to two-person interactions, we propose a novel social motion representation that supports tokenizing the motions of an arbitrary number of individuals.<n>This alignment enables the model to leverage rich, pretrained linguistic knowledge to better understand and reason about human social behaviors.
• Score: 30.068879125411726
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Human interactions in everyday life are inherently social, involving engagements with diverse individuals across various contexts. Modeling these social interactions is fundamental to a wide range of real-world applications. In this paper, we introduce SocialGen, the first unified motion-language model capable of modeling interaction behaviors among varying numbers of individuals, to address this crucial yet challenging problem. Unlike prior methods that are limited to two-person interactions, we propose a novel social motion representation that supports tokenizing the motions of an arbitrary number of individuals and aligning them with the language space. This alignment enables the model to leverage rich, pretrained linguistic knowledge to better understand and reason about human social behaviors. To tackle the challenges of data scarcity, we curate a comprehensive multi-human interaction dataset, SocialX, enriched with textual annotations. Leveraging this dataset, we establish the first comprehensive benchmark for multi-human interaction tasks. Our method achieves state-of-the-art performance across motion-language tasks, setting a new standard for multi-human interaction modeling.

Related papers

• SocioVerse: A World Model for Social Simulation Powered by LLM Agents and A Pool of 10 Million Real-World Users [70.02370111025617]
  We introduce SocioVerse, an agent-driven world model for social simulation. Our framework features four powerful alignment components and a user pool of 10 million real individuals. Results demonstrate that SocioVerse can reflect large-scale population dynamics while ensuring diversity, credibility, and representativeness.
  arXiv  Detail & Related papers  (2025-04-14T12:12:52Z)
• SocialGesture: Delving into Multi-person Gesture Understanding [24.407821757676615]
  SocialGesture is the first large-scale dataset specifically designed for multi-person gesture analysis. SocialGesture features a diverse range of natural scenarios and supports multiple gesture analysis tasks.
  arXiv  Detail & Related papers  (2025-04-03T03:21:06Z)
• A Survey on Human Interaction Motion Generation [17.364321072349004]
  Humans inhabit a world defined by interactions -- with other humans, objects, and environments. Interactive movements convey our relationships with our surroundings and demonstrate how we perceive and communicate with the real world. replicating these interaction behaviors in digital systems has emerged as an important topic for applications in robotics, virtual reality, and animation.
  arXiv  Detail & Related papers  (2025-03-17T02:55:10Z)
• PersLLM: A Personified Training Approach for Large Language Models [66.16513246245401]
  We propose PersLLM, integrating psychology-grounded principles of personality: social practice, consistency, and dynamic development. We incorporate personality traits directly into the model parameters, enhancing the model's resistance to induction, promoting consistency, and supporting the dynamic evolution of personality.
  arXiv  Detail & Related papers  (2024-07-17T08:13:22Z)
• in2IN: Leveraging individual Information to Generate Human INteractions [29.495166514135295]
  We introduce in2IN, a novel diffusion model for human-human motion generation conditioned on individual descriptions. We also propose DualMDM, a model composition technique that combines the motions generated with in2IN and the motions generated by a single-person motion prior pre-trained on HumanML3D.
  arXiv  Detail & Related papers  (2024-04-15T17:59:04Z)
• JRDB-Social: A Multifaceted Robotic Dataset for Understanding of Context and Dynamics of Human Interactions Within Social Groups [8.415759777703125]
  JRDB-Social fills gaps in human understanding across diverse indoor and outdoor social contexts. This dataset aims to enhance our grasp of human social dynamics for robotic applications.
  arXiv  Detail & Related papers  (2024-04-06T00:33:39Z)
• THOR: Text to Human-Object Interaction Diffusion via Relation Intervention [51.02435289160616]
  We propose a novel Text-guided Human-Object Interaction diffusion model with Relation Intervention (THOR) In each diffusion step, we initiate text-guided human and object motion and then leverage human-object relations to intervene in object motion. We construct Text-BEHAVE, a Text2HOI dataset that seamlessly integrates textual descriptions with the currently largest publicly available 3D HOI dataset.
  arXiv  Detail & Related papers  (2024-03-17T13:17:25Z)
• Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations [20.848802791989307]
  We introduce three new challenges to model the fine-grained dynamics between multiple people: speaking target identification, pronoun coreference resolution, and mentioned player prediction. We propose a novel multimodal baseline that leverages densely aligned language-visual representations by synchronizing visual features with their corresponding utterances. Experiments demonstrate the effectiveness of the proposed approach with densely aligned multimodal representations in modeling fine-grained social interactions.
  arXiv  Detail & Related papers  (2024-03-04T14:46:58Z)
• Face-to-Face Contrastive Learning for Social Intelligence Question-Answering [55.90243361923828]
  multimodal methods have set the state of the art on many tasks, but have difficulty modeling the complex face-to-face conversational dynamics. We propose Face-to-Face Contrastive Learning (F2F-CL), a graph neural network designed to model social interactions. We experimentally evaluated the challenging Social-IQ dataset and show state-of-the-art results.
  arXiv  Detail & Related papers  (2022-07-29T20:39:44Z)
• PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception [50.551003004553806]
  We create a dataset of physically-grounded abstract social events, PHASE, that resemble a wide range of real-life social interactions. Phase is validated with human experiments demonstrating that humans perceive rich interactions in the social events. As a baseline model, we introduce a Bayesian inverse planning approach, SIMPLE, which outperforms state-of-the-art feed-forward neural networks.
  arXiv  Detail & Related papers  (2021-03-02T18:44:57Z)
• Human Trajectory Forecasting in Crowds: A Deep Learning Perspective [89.4600982169]
  We present an in-depth analysis of existing deep learning-based methods for modelling social interactions. We propose two knowledge-based data-driven methods to effectively capture these social interactions. We develop a large scale interaction-centric benchmark TrajNet++, a significant yet missing component in the field of human trajectory forecasting.
  arXiv  Detail & Related papers  (2020-07-07T17:19:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.