GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions

• URL: http://arxiv.org/abs/2605.15764v1
• Date: Fri, 15 May 2026 09:24:41 GMT
• Title: GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions
• Abstract summary: We introduce GRASP, a large-scale social reasoning dataset that connects high-level social QA with fine-grained gaze and deictic gesture events.<n>GraSP contains 290K question--answer pairs over 46K videos totaling 749 hours, organized by a 16-category taxonomy spanning gaze, gesture, and joint gaze--gesture reasoning.<n>We propose Social Grounding Reward (SGR), a learning signal that uses these social events to encourage models to reason about the participants involved in each interaction.
• Score: 44.26597369595731
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Understanding social interactions requires reasoning over subtle non-verbal cues, yet current multimodal large language models (MLLMs) often fail to identify who interacts with whom in multi-person videos. We introduce GRASP, a large-scale social reasoning dataset that connects high-level social QA with fine-grained gaze and deictic gesture events. GRASP contains 290K question--answer pairs over 46K videos totaling 749 hours, organized by a 16-category taxonomy spanning gaze, gesture, and joint gaze--gesture reasoning, together with GRASP-Bench for evaluation. Unlike prior resources that focus on either isolated cues or high-level social QA, GRASP builds questions from identity-consistent gaze trajectories, deictic gestures, and their joint compositions into social events. Moreover, we propose Social Grounding Reward (SGR), a learning signal that uses these social events to encourage models to reason about the participants involved in each interaction. Experiments show that SGR improves performance on GRASP-Bench while maintaining zero-shot performance on related social video QA benchmarks.

Related papers

• SocialDirector: Training-Free Social Interaction Control for Multi-Person Video Generation [25.88676013839077]
  Video generation has advanced rapidly, producing videos from text or image prompts.<n>Film production and social robotics increasingly demand multi-person videos with rich social interactions.<n>Existing models offer no explicit control over interactions, such as who performs which action, when it occurs, and toward whom it is directed.<n>We present SocialDirector, a training-free interaction controller that enhances the generation model by cross-attention maps.
  arXiv  Detail & Related papers  (2026-05-11T07:01:38Z)
• SIV-Bench: A Video Benchmark for Social Interaction Understanding and Reasoning [53.16179295245888]
  We introduce SIV-Bench, a novel video benchmark for evaluating the capabilities of Multimodal Large Language Models (MLLMs) across Social Scene Understanding (SSU), Social State Reasoning (SSR), and Social Dynamics Prediction (SDP)<n>SIV-Bench features 2,792 video clips and 8,792 meticulously generated question-answer pairs derived from a human-LLM collaborative pipeline.<n>It also includes a dedicated setup for analyzing the impact of different textual cues-original on-screen text, added dialogue, or no text.
  arXiv  Detail & Related papers  (2025-06-05T05:51:35Z)
• SocialGesture: Delving into Multi-person Gesture Understanding [24.407821757676615]
  SocialGesture is the first large-scale dataset specifically designed for multi-person gesture analysis.<n>SocialGesture features a diverse range of natural scenarios and supports multiple gesture analysis tasks.
  arXiv  Detail & Related papers  (2025-04-03T03:21:06Z)
• Social Genome: Grounded Social Reasoning Abilities of Multimodal Models [61.88413918026431]
  Social reasoning abilities are crucial for AI systems to interpret and respond to multimodal human communication and interaction within social contexts.<n>We introduce SOCIAL GENOME, the first benchmark for fine-grained, grounded social reasoning abilities of multimodal models.
  arXiv  Detail & Related papers  (2025-02-21T00:05:40Z)
• Love in Action: Gamifying Public Video Cameras for Fostering Social Relationships in Real World [15.049706359599666]
  We create "Love in Action" (LIA), a body language-based social game utilizing video cameras installed in public spaces. A two-week field study involving 27 participants shows significant improvements in their social friendships. Users experiences are investigated to highlight the potential of public video cameras as a novel communication medium for socializing in public spaces.
  arXiv  Detail & Related papers  (2024-10-31T02:38:40Z)
• From a Social Cognitive Perspective: Context-aware Visual Social Relationship Recognition [59.57095498284501]
  We propose a novel approach that recognizes textbfContextual textbfSocial textbfRelationships (textbfConSoR) from a social cognitive perspective. We construct social-aware descriptive language prompts with social relationships for each image. Impressively, ConSoR outperforms previous methods with a 12.2% gain on the People-in-Social-Context (PISC) dataset and a 9.8% increase on the People-in-Photo-Album (PIPA) benchmark.
  arXiv  Detail & Related papers  (2024-06-12T16:02:28Z)
• Face-to-Face Contrastive Learning for Social Intelligence Question-Answering [55.90243361923828]
  multimodal methods have set the state of the art on many tasks, but have difficulty modeling the complex face-to-face conversational dynamics. We propose Face-to-Face Contrastive Learning (F2F-CL), a graph neural network designed to model social interactions. We experimentally evaluated the challenging Social-IQ dataset and show state-of-the-art results.
  arXiv  Detail & Related papers  (2022-07-29T20:39:44Z)
• PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception [50.551003004553806]
  We create a dataset of physically-grounded abstract social events, PHASE, that resemble a wide range of real-life social interactions. Phase is validated with human experiments demonstrating that humans perceive rich interactions in the social events. As a baseline model, we introduce a Bayesian inverse planning approach, SIMPLE, which outperforms state-of-the-art feed-forward neural networks.
  arXiv  Detail & Related papers  (2021-03-02T18:44:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.