WaveSync: Constrained Wavefront Optimization for Synchronized Co-Speech Gestures in Humanoid Robots

• URL: http://arxiv.org/abs/2606.16600v1
• Date: Mon, 15 Jun 2026 11:47:30 GMT
• Title: WaveSync: Constrained Wavefront Optimization for Synchronized Co-Speech Gestures in Humanoid Robots
• Abstract summary: Expressive co-speech gestures are crucial for natural human-robot interaction, but generating them on physical humanoid robots is difficult.<n>We present textbfWaveSync, a hybrid framework in which a Large Language Model decomposes dialogue responses into structured semantic schemas.<n>We show that our method achieves high synchronization accuracy and outperforms three baselines in both objective and subjective evaluations.
• Score: 2.0424038686449006
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Expressive co-speech gestures are crucial for natural human-robot interaction, but generating them on physical humanoid robots is difficult because gesture strokes must align with speech emphasis while satisfying strict kinematic and dynamic constraints. Unlike virtual avatars, humanoid robots cannot freely execute rapid or overlapping motions, making word-level synchronization and hardware-safe motion planning a coupled problem. We present \textbf{WaveSync}, a hybrid framework in which a Large Language Model decomposes dialogue responses into structured semantic schemas and assigns per-word importance weights, constructing a continuous Semantic Importance Wave. Gesture trajectories are shaped through Dynamic Movement Primitives, enforcing kinematic feasibility while enhancing expressiveness. A Wavefront Optimization stage aligns peak-to-peak gesture-speech synchronization and resolves residual kinematic violations through gesture-duration compression and forward propagation. Experimental evaluation based on five dialogue scenarios shows that our method achieves high synchronization accuracy and outperforms three baselines in both objective and subjective evaluations. Each component in WaveSync plays a necessary role in producing gestures that are expressive, semantically grounded, and kinematically compliant. The code, resources, and videos are available at \href{https://github.com/pairs-lab/WaveSync}{WaveSync}

Related papers

• RoboGesture: Real-Time Semantic-aligned Co-Speech Gestures Generation for Humanoid Interaction [33.04319504310729]
  RoboGesture is a robot-centric framework that co-designs data, modeling, and control to power a complete interactive human-humanoid system.<n>We first establish the RoboGesture dataset featuring over 300 gesture categories and develop an automated pipeline to synthesize large-scale audio-motion pairs.<n>Experiments on a Unitree G1 humanoid demonstrate that RoboGesture generates safer, more rhythmic, and more semantically appropriate responses compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-08-27T02:13:46Z)
• ReFree: Towards Realistic Co-Speech Video Generation via Reward-Free RL and Multilevel Speech Guidance [50.482989497576476]
  Speech-driven talking character animation seeks to generate life-like portrait videos that convey natural conversation behavior.<n>Existing approaches typically trade off precise phoneme-to-lip synchronization against dynamic facial expressions and head motion.<n>We propose ReFree-S2V, a flow-matching speech-to-portrait animation framework that builds upon a pretrained video generation model.
  arXiv  Detail & Related papers  (2026-06-11T13:00:58Z)
• Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation [50.411841997631484]
  We present Unison, a unified framework that promotes coherence across the motion, speech, and sound modalities.<n>We show that Unison achieves state-of-the-art performance in both audio perceptual quality and cross-modal synchronization.
  arXiv  Detail & Related papers  (2026-05-09T06:32:54Z)
• MIBURI: Towards Expressive Interactive Gesture Synthesis [62.45332399212876]
  Embodied Conversational Agents (ECAs) aim to emulate human face-to-face interaction through speech, gestures, and facial expressions.<n>Existing solutions for ECAs produce rigid, low-diversity motions that are unsuitable for human-like interaction.<n>We present MIBURI, the first online, causal framework for generating expressive full-body gestures and facial expressions synchronized with real-time spoken dialogue.
  arXiv  Detail & Related papers  (2026-03-03T18:59:51Z)
• Real-Time Synchronized Interaction Framework for Emotion-Aware Humanoid Robots [0.5156484100374058]
  We present a real-time framework for NAO robots that synchronizes speech prosody with full-body gestures.<n>By enabling seamless sensorimotor coordination, this framework advances the deployment of context-aware social robots.
  arXiv  Detail & Related papers  (2026-01-24T03:53:09Z)
• OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulation [29.41106195298283]
  Existing video avatar models can produce fluid human animations, yet they struggle to move beyond mere physical likeness to capture a character's authentic essence.<n>textbfwe propose a framework designed to generate character animations that are not only physically plausible but also semantically coherent and expressive.
  arXiv  Detail & Related papers  (2025-08-26T17:15:26Z)
• AsynFusion: Towards Asynchronous Latent Consistency Models for Decoupled Whole-Body Audio-Driven Avatars [71.90109867684025]
  Whole-body audio-driven avatar pose and expression generation is a critical task for creating lifelike digital humans.<n>We propose AsynFusion, a novel framework that leverages diffusion transformers to achieve cohesive expression and gesture synthesis.<n>AsynFusion achieves state-of-the-art performance in generating real-time, synchronized whole-body animations.
  arXiv  Detail & Related papers  (2025-05-21T03:28:53Z)
• HoloGest: Decoupled Diffusion and Motion Priors for Generating Holisticly Expressive Co-speech Gestures [8.50717565369252]
  HoleGest is a novel neural network framework for automatic generation of high-quality, expressive co-speech gestures.<n>Our system learns a robust prior with low audio dependency and high motion reliance, enabling stable global motion and detailed finger movements.<n>Our model achieves a level of realism close to the ground truth, providing an immersive user experience.
  arXiv  Detail & Related papers  (2025-03-17T14:42:31Z)
• Cosh-DiT: Co-Speech Gesture Video Synthesis via Hybrid Audio-Visual Diffusion Transformers [58.86974149731874]
  Cosh-DiT is a Co-speech gesture video system with hybrid Diffusion Transformers.<n>We introduce an audio Diffusion Transformer to synthesize expressive gesture dynamics synchronized with speech rhythms.<n>For realistic video synthesis conditioned on the generated speech-driven motion, we design a visual Diffusion Transformer.
  arXiv  Detail & Related papers  (2025-03-13T01:36:05Z)
• QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation [8.604430209445695]
  Speech-driven gesture generation is highly challenging due to the random jitters of human motion. We introduce a novel quantization-based and phase-guided motion-matching framework. Our method outperforms recent approaches on speech-driven gesture generation.
  arXiv  Detail & Related papers  (2023-05-18T16:31:25Z)
• Freeform Body Motion Generation from Speech [53.50388964591343]
  Body motion generation from speech is inherently difficult due to the non-deterministic mapping from speech to body motions. We introduce a novel freeform motion generation model (FreeMo) by equipping a two-stream architecture. Experiments demonstrate the superior performance against several baselines.
  arXiv  Detail & Related papers  (2022-03-04T13:03:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.