SocialDirector: Training-Free Social Interaction Control for Multi-Person Video Generation

• URL: http://arxiv.org/abs/2605.10079v1
• Date: Mon, 11 May 2026 07:01:38 GMT
• Title: SocialDirector: Training-Free Social Interaction Control for Multi-Person Video Generation
• Abstract summary: Video generation has advanced rapidly, producing videos from text or image prompts.<n>Film production and social robotics increasingly demand multi-person videos with rich social interactions.<n>Existing models offer no explicit control over interactions, such as who performs which action, when it occurs, and toward whom it is directed.<n>We present SocialDirector, a training-free interaction controller that enhances the generation model by cross-attention maps.
• Score: 25.88676013839077
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video generation has advanced rapidly, producing photorealistic videos from text or image prompts. Meanwhile, film production and social robotics increasingly demand multi-person videos with rich social interactions, including conversations, gestures, and coordinated actions. However, existing models offer no explicit control over interactions, such as who performs which action, when it occurs, and toward whom it is directed. This often results in wrong person performing unintended actions (actor-action mismatch), disordered social dynamics, and wrong action targets. To address these challenges, we present SocialDirector, a training-free interaction controller that enhances the generation model by modulating cross-attention maps. SocialDirector contains two modules: Social Actor Masking and Directional Reweighting. Social Actor Masking constrains each person's visual tokens to attend only to their own textual descriptions via a spatiotemporal mask, avoiding actor-action mismatch and disordered social dynamics. Directional Reweighting amplifies attention to directional words (e.g., "leftward", "right"), leading each action towards its intended target. To evaluate generated social interactions, we annotate existing datasets with interaction descriptions and build a fully automated evaluation pipeline powered by open-source VLMs. Experiments on different video generation models show that SocialDirector significantly improves interaction fidelity and approaches the upper bound set by real videos.

Related papers

• Masked Visual Actions for Unified World Modeling [96.12988421978463]
  We introduce Masked Visual Actions, a pixel-space control interface that expresses action as a partially revealed trajectory of an arbitrary entity in a video.<n>Revealing robot motion makes the model act as a forward dynamics model that predicts the scene's response to low-level robot actions, while revealing desired object motion makes the same model recover robot behavior consistent with that outcome.
  arXiv  Detail & Related papers  (2026-07-21T17:59:11Z)
• Resonant Minds: Closed-Loop Social Avatars with Theory of Mind [16.880605576970538]
  We propose a closed-loop dual-agent framework integrating perception, social reasoning, and expression into a continuous interaction cycle.<n>The perception module analyzes partners' multimodal behaviors from video, while the social reasoning module infers hidden mental states through Theory of Mind.<n>The expression module then generates emotion-controllable dual-agent videos both speaker speech and expression alongside listener reactive behaviors.
  arXiv  Detail & Related papers  (2026-06-04T09:03:43Z)
• ActionParty: Multi-Subject Action Binding in Generative Video Games [117.52562594944679]
  ActionParty is a controllable multi-subject world model for generative video games.<n>We demonstrate the first video world model capable of controlling up to seven players simultaneously across 46 diverse environments.
  arXiv  Detail & Related papers  (2026-04-02T17:59:58Z)
• Developing Neural Network-Based Gaze Control Systems for Social Robots [0.0]
  This study aims to develop an empirical motion-time pattern for human gaze behavior in various social situations.<n>Deep learning models, specifically Long Short-Term Memory (LSTM) and Transformers, were used to analyze and predict gaze patterns.
  arXiv  Detail & Related papers  (2026-02-11T15:30:39Z)
• InterAct: A Large-Scale Dataset of Dynamic, Expressive and Interactive Activities between Two People in Daily Scenarios [40.42003202491803]
  We propose to simultaneously model two people's activities, and target objective-driven, dynamic, and semantically consistent interactions.<n>We capture a new multi-modal dataset dubbed InterAct composed of 241 motion sequences.<n>InterAct contains diverse and complex motions of individuals and interesting and relatively long-term interaction patterns barely seen before.
  arXiv  Detail & Related papers  (2025-09-06T15:36:47Z)
• Taming Transformer for Emotion-Controllable Talking Face Generation [61.835295250047196]
  We propose a novel method to tackle the emotion-controllable talking face generation task discretely.<n>Specifically, we employ two pre-training strategies to disentangle audio into independent components and quantize videos into combinations of visual tokens.<n>We conduct experiments on the MEAD dataset that controls the emotion of videos conditioned on multiple emotional audios.
  arXiv  Detail & Related papers  (2025-08-20T02:16:52Z)
• Love in Action: Gamifying Public Video Cameras for Fostering Social Relationships in Real World [15.049706359599666]
  We create "Love in Action" (LIA), a body language-based social game utilizing video cameras installed in public spaces. A two-week field study involving 27 participants shows significant improvements in their social friendships. Users experiences are investigated to highlight the potential of public video cameras as a novel communication medium for socializing in public spaces.
  arXiv  Detail & Related papers  (2024-10-31T02:38:40Z)
• Social-MAE: Social Masked Autoencoder for Multi-person Motion Representation Learning [16.85316898267404]
  We introduce Social-MAE, a transformer-based masked autoencoder framework for multi-person human motion data. The framework uses masked modeling to pre-train the encoder to reconstruct masked human joint trajectories. It achieves the state-of-the-art results on various high-level social tasks, including multi-person pose forecasting, social grouping, and social action understanding.
  arXiv  Detail & Related papers  (2024-04-08T14:54:54Z)
• Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training [69.54948297520612]
  Learning a generalist embodied agent poses challenges, primarily stemming from the scarcity of action-labeled robotic datasets. We introduce a novel framework to tackle these challenges, which leverages a unified discrete diffusion to combine generative pre-training on human videos and policy fine-tuning on a small number of action-labeled robot videos. Our method generates high-fidelity future videos for planning and enhances the fine-tuned policies compared to previous state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-02-22T09:48:47Z)
• Face-to-Face Contrastive Learning for Social Intelligence Question-Answering [55.90243361923828]
  multimodal methods have set the state of the art on many tasks, but have difficulty modeling the complex face-to-face conversational dynamics. We propose Face-to-Face Contrastive Learning (F2F-CL), a graph neural network designed to model social interactions. We experimentally evaluated the challenging Social-IQ dataset and show state-of-the-art results.
  arXiv  Detail & Related papers  (2022-07-29T20:39:44Z)
• SSAGCN: Social Soft Attention Graph Convolution Network for Pedestrian Trajectory Prediction [59.064925464991056]
  We propose one new prediction model named Social Soft Attention Graph Convolution Network (SSAGCN) SSAGCN aims to simultaneously handle social interactions among pedestrians and scene interactions between pedestrians and environments. Experiments on public available datasets prove the effectiveness of SSAGCN and have achieved state-of-the-art results.
  arXiv  Detail & Related papers  (2021-12-05T01:49:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.