ReactMotion: Generating Reactive Listener Motions from Speaker Utterance

• URL: http://arxiv.org/abs/2603.15083v1
• Date: Mon, 16 Mar 2026 10:37:42 GMT
• Title: ReactMotion: Generating Reactive Listener Motions from Speaker Utterance
• Abstract summary: Reactive Listener Motion Generation from Speaker Utterance aims to generate naturalistic listener body motions that appropriately respond to a speaker's utterance.<n>We present ReactMotionNet, a large-scale dataset that pairs speaker utterances with multiple candidate listener motions annotated with varying degrees of appropriateness.<n>We propose ReactMotion, a unified generative framework that jointly models text, audio, emotion, and motion, and is trained with preference-based objectives to encourage both appropriate and diverse listener responses.
• Score: 93.89117265072464
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In this paper, we introduce a new task, Reactive Listener Motion Generation from Speaker Utterance, which aims to generate naturalistic listener body motions that appropriately respond to a speaker's utterance. However, modeling such nonverbal listener behaviors remains underexplored and challenging due to the inherently non-deterministic nature of human reactions. To facilitate this task, we present ReactMotionNet, a large-scale dataset that pairs speaker utterances with multiple candidate listener motions annotated with varying degrees of appropriateness. This dataset design explicitly captures the one-to-many nature of listener behavior and provides supervision beyond a single ground-truth motion. Building on this dataset design, we develop preference-oriented evaluation protocols tailored to evaluate reactive appropriateness, where conventional motion metrics focusing on input-motion alignment ignore. We further propose ReactMotion, a unified generative framework that jointly models text, audio, emotion, and motion, and is trained with preference-based objectives to encourage both appropriate and diverse listener responses. Extensive experiments show that ReactMotion outperforms retrieval baselines and cascaded LLM-based pipelines, generating more natural, diverse, and appropriate listener motions.

Related papers

• MultiAct: Text-to-Motion Generation from Composite Text via Tailored Attention Guidance [54.131431576597514]
  MultiAct is an unpaired, inference-time framework for compositional text-to-motion synthesis.<n>Our method counteracts semantic collapse by adaptively amplifying cross-attention scores associated with underrepresented prompt components.
  arXiv  Detail & Related papers  (2026-05-29T07:14:48Z)
• InterDyad: Interactive Dyadic Speech-to-Video Generation by Querying Intermediate Visual Guidance [20.740979380270126]
  We propose InterDyad, a synthesis framework for interactive dyadic dynamics.<n>We first design an Interactivity that achieves video reenactment based on identity-agnostic motion priors extracted from reference videos.<n>By leveraging a Multimodal Large Language Model (MLLM), our framework is able to distill linguistic intent from audio to dictate the precise timing and appropriateness of reactions.<n> Comprehensive experiments demonstrate that InterDyad significantly outperforms state-of-the-art methods in producing natural and contextually grounded two-person interactions.
  arXiv  Detail & Related papers  (2026-03-24T12:27:52Z)
• MoReact: Generating Reactive Motion from Textual Descriptions [57.642436102978245]
  MoReact is a diffusion-based method designed to disentangle the generation of global trajectories and local motions sequentially.<n>Our experiments, utilizing data adapted from a two-person motion dataset, demonstrate the efficacy of our approach.
  arXiv  Detail & Related papers  (2025-09-28T14:31:41Z)
• Aligning Spoken Dialogue Models from User Interactions [55.192134724622235]
  We propose a novel preference alignment framework to improve spoken dialogue models on realtime conversations from user interactions.<n>We create a dataset of more than 150,000 preference pairs from raw multi-turn speech conversations annotated with AI feedback.<n>Our findings shed light on the importance of a well-calibrated balance among various dynamics, crucial for natural real-time speech dialogue systems.
  arXiv  Detail & Related papers  (2025-06-26T16:45:20Z)
• VividListener: Expressive and Controllable Listener Dynamics Modeling for Multi-Modal Responsive Interaction [31.307004436877587]
  We propose VividListener, a novel framework enabling fine-grained, expressive and controllable listener dynamics modeling.<n>VividListener achieves state-of-the-art performance, realizing expressive and controllable listener dynamics.
  arXiv  Detail & Related papers  (2025-04-30T15:05:12Z)
• Dyadic Interaction Modeling for Social Behavior Generation [6.626277726145613]
  We present an effective framework for creating 3D facial motions in dyadic interactions. The heart of our framework is Dyadic Interaction Modeling (DIM), a pre-training approach. Experiments demonstrate the superiority of our framework in generating listener motions.
  arXiv  Detail & Related papers  (2024-03-14T03:21:33Z)
• CustomListener: Text-guided Responsive Interaction for User-friendly Listening Head Generation [9.741109135330262]
  Listening head generation aims to synthesize a non-verbal responsive listener head by modeling the correlation between the speaker and the listener in dynamic conversion. We propose a user-friendly framework called CustomListener to realize the free-form text prior guided listener generation.
  arXiv  Detail & Related papers  (2024-03-01T04:31:56Z)
• Emotional Listener Portrait: Realistic Listener Motion Simulation in Conversation [50.35367785674921]
  Listener head generation centers on generating non-verbal behaviors of a listener in reference to the information delivered by a speaker. A significant challenge when generating such responses is the non-deterministic nature of fine-grained facial expressions during a conversation. We propose the Emotional Listener Portrait (ELP), which treats each fine-grained facial motion as a composition of several discrete motion-codewords. Our ELP model can not only automatically generate natural and diverse responses toward a given speaker via sampling from the learned distribution but also generate controllable responses with a predetermined attitude.
  arXiv  Detail & Related papers  (2023-09-29T18:18:32Z)
• Can Language Models Learn to Listen? [96.01685069483025]
  We present a framework for generating appropriate facial responses from a listener in dyadic social interactions based on the speaker's words. Our approach autoregressively predicts a response of a listener: a sequence of listener facial gestures, quantized using a VQ-VAE. We show that our generated listener motion is fluent and reflective of language semantics through quantitative metrics and a qualitative user study.
  arXiv  Detail & Related papers  (2023-08-21T17:59:02Z)
• Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion [89.01668641930206]
  We present a framework for modeling interactional communication in dyadic conversations. We autoregressively output multiple possibilities of corresponding listener motion. Our method organically captures the multimodal and non-deterministic nature of nonverbal dyadic interactions.
  arXiv  Detail & Related papers  (2022-04-18T17:58:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.