RoboGesture: Real-Time Semantic-aligned Co-Speech Gestures Generation for Humanoid Interaction

• URL: http://arxiv.org/abs/2608.28693v1
• Date: Thu, 27 Aug 2026 02:13:46 GMT
• Title: RoboGesture: Real-Time Semantic-aligned Co-Speech Gestures Generation for Humanoid Interaction
• Abstract summary: RoboGesture is a robot-centric framework that co-designs data, modeling, and control to power a complete interactive human-humanoid system.<n>We first establish the RoboGesture dataset featuring over 300 gesture categories and develop an automated pipeline to synthesize large-scale audio-motion pairs.<n>Experiments on a Unitree G1 humanoid demonstrate that RoboGesture generates safer, more rhythmic, and more semantically appropriate responses compared to state-of-the-art baselines.
• Score: 33.04319504310729
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Enabling humanoid robots to respond to human speech with synchronized and semantically meaningful gestures is fundamental to natural human-robot interaction. However, this task faces three critical barriers: the scarcity of semantically rich datasets, the "modality eclipse" where models ignore audio cues in favor of kinematic inertia, and the sim-to-real gap regarding physical safety. We propose RoboGesture, a robot-centric framework that co-designs data, modeling, and control to power a complete interactive human-humanoid system in which the robot listens, responds, and gestures in real time. We first establish the RoboGesture dataset featuring over 300 gesture categories and develop an automated pipeline to synthesize large-scale collision-free, robot-specific audio-motion pairs. Our architecture features a Hierarchical Semantic-Acoustic Aligner that extracts multi-granular prosodic and semantic cues directly from raw audio tokens. These cues drive a Streaming Conditional Motion Generator based on a diffusion transformer with conditional flow matching. To ensure high responsiveness, we introduce Anti-Inertia CFG Masking, which prevents the model from collapsing into repetitive historical patterns by compelling it to proactively mine control signals from the audio modality. Finally, an MPC-based safety filter ensures real-time, collision-free execution on physical hardware. Experiments on a Unitree G1 humanoid demonstrate that RoboGesture generates safer, more rhythmic, and more semantically appropriate responses compared to state-of-the-art baselines.

Related papers

• RGB-D Video Generation for Improving Human-to-Robot Object Handover Prediction [51.76737991200891]
  Human-to-robot (H2R) object handover is a fundamental capability for human-robot collaboration, yet progress is hindered by the scarcity of large-scale, human-centric datasets.<n>We introduce Hand2Bot, an RGB-D video dataset that provides rich contextual information such as body posture and facial expressions.<n>We propose PassGen, a generative pipeline that synthesizes realistic handover sequences while ensuring hand-object consistency.
  arXiv  Detail & Related papers  (2026-08-13T09:55:59Z)
• Semantic Audio-driven Understanding for Dynamic Humanoid Whole Body Control [0.0]
  We introduce a novel multi-modal orchestration framework for semantic audio-driven humanoid control.<n>The system processes continuous audio streams and routes them into music or speech branches.<n>Music input is handled via audio fingerprinting and semantic embeddings to retrieve track identity and temporal alignment.<n>Speech input is grounded into a discrete library of imitation-learned skills, enabling direct human-robot interaction.
  arXiv  Detail & Related papers  (2026-07-15T14:24:00Z)
• LLM-Powered Interactive Robotic Action Synthesis from Multimodal Speech, Gestures, and Music [1.7188280334580195]
  This paper introduces a novel framework that synthesizes complex robotic actions from a rich tapestry of multimodal human inputs.<n>Our system architecture integrates a speech transcription model, a gesture recognition module, and a signal processing pipeline for beat detection.<n>The framework has ability to interpret and fuse semantic commands from speech, deictic information from gestures, and rhythmic cues from music.
  arXiv  Detail & Related papers  (2026-06-30T05:35:26Z)
• TEXEDO : Test Time Scaling for Controller-aware Language-conditioned Humanoid Motion Generation [48.72797830886802]
  We introduce TEXEDO, a test-time scaling framework for humanoid motion generation.<n>It improves motion quality without requiring a stronger underlying generator.<n>We show it consistently improves both tracking fidelity and text alignment.
  arXiv  Detail & Related papers  (2026-06-22T08:14:35Z)
• WaveSync: Constrained Wavefront Optimization for Synchronized Co-Speech Gestures in Humanoid Robots [2.0424038686449006]
  Expressive co-speech gestures are crucial for natural human-robot interaction, but generating them on physical humanoid robots is difficult.<n>We present textbfWaveSync, a hybrid framework in which a Large Language Model decomposes dialogue responses into structured semantic schemas.<n>We show that our method achieves high synchronization accuracy and outperforms three baselines in both objective and subjective evaluations.
  arXiv  Detail & Related papers  (2026-06-15T11:47:30Z)
• MIBURI: Towards Expressive Interactive Gesture Synthesis [62.45332399212876]
  Embodied Conversational Agents (ECAs) aim to emulate human face-to-face interaction through speech, gestures, and facial expressions.<n>Existing solutions for ECAs produce rigid, low-diversity motions that are unsuitable for human-like interaction.<n>We present MIBURI, the first online, causal framework for generating expressive full-body gestures and facial expressions synchronized with real-time spoken dialogue.
  arXiv  Detail & Related papers  (2026-03-03T18:59:51Z)
• Semantic Co-Speech Gesture Synthesis and Real-Time Control for Humanoid Robots [5.531678625546847]
  We present an innovative end-to-end framework for synthesizing semantically meaningful co-speech gestures and deploying them in real-time on a humanoid robot.<n>Our core innovation lies in the meticulous integration of a semantics-aware gesture synthesis module.<n>We demonstrate that our combined system produces semantically appropriate and rhythmically coherent gestures.
  arXiv  Detail & Related papers  (2025-12-19T02:55:10Z)
• XR-DT: Extended Reality-Enhanced Digital Twin for Agentic Mobile Robots [10.083050242188422]
  This paper presents XR-DT, an eXtended Reality-enhanced Digital Twin framework for agentic mobile robots.<n>By embedding human intention, environmental dynamics, and robot cognition into the XR-DT framework, our system enables interpretable, trustworthy, and adaptive HRI.
  arXiv  Detail & Related papers  (2025-12-04T21:49:14Z)
• From Language to Locomotion: Retargeting-free Humanoid Control via Motion Latent Guidance [55.31807046722006]
  Existing language-guided humanoid pipelines are cumbersome and untrustworthy.<n>We present RoboGhost, a language-free framework that conditions humanoid policies on language-grounded motion latents.<n>We show that RoboGhost substantially reduces deployment latency, improves success rates and tracking precision, and produces smooth, semantically aligned humanoids.
  arXiv  Detail & Related papers  (2025-10-16T17:57:47Z)
• OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction [76.44108003274955]
  A dominant paradigm for teaching humanoid robots complex skills is to retarget human motions as kinematic references to train reinforcement learning policies.<n>We introduce OmniRetarget, an interaction-preserving data generation engine based on an interaction mesh.<n>By minimizing the Laplacian deformation between the human and robot meshes, OmniRetarget generates kinematically feasible trajectories.
  arXiv  Detail & Related papers  (2025-09-30T17:59:02Z)
• RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation [77.41969287400977]
  This paper presents textbfRobotScript, a platform for a deployable robot manipulation pipeline powered by code generation. We also present a benchmark for a code generation benchmark for robot manipulation tasks in free-form natural language. We demonstrate the adaptability of our code generation framework across multiple robot embodiments, including the Franka and UR5 robot arms.
  arXiv  Detail & Related papers  (2024-02-22T15:12:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.