Do You Have Freestyle? Expressive Humanoid Locomotion via Audio Control

• URL: http://arxiv.org/abs/2512.23650v2
• Date: Sun, 04 Jan 2026 07:49:24 GMT
• Title: Do You Have Freestyle? Expressive Humanoid Locomotion via Audio Control
• Abstract summary: Current humanoid robots lack expressive improvisational capabilities, confined to predefined motions or sparse commands.<n>We propose RoboPerform, the first unified audio-to-locomotion framework that can directly generate music-driven dance and speech-driven co-speech gestures from audio.<n>RoboPerform achieves a ResMoE policy for adapting to diverse motion patterns and a diffusion-based student policy for audio style injection.
• Score: 52.83779852397341
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Humans intuitively move to sound, but current humanoid robots lack expressive improvisational capabilities, confined to predefined motions or sparse commands. Generating motion from audio and then retargeting it to robots relies on explicit motion reconstruction, leading to cascaded errors, high latency, and disjointed acoustic-actuation mapping. We propose RoboPerform, the first unified audio-to-locomotion framework that can directly generate music-driven dance and speech-driven co-speech gestures from audio. Guided by the core principle of "motion = content + style", the framework treats audio as implicit style signals and eliminates the need for explicit motion reconstruction. RoboPerform integrates a ResMoE teacher policy for adapting to diverse motion patterns and a diffusion-based student policy for audio style injection. This retargeting-free design ensures low latency and high fidelity. Experimental validation shows that RoboPerform achieves promising results in physical plausibility and audio alignment, successfully transforming robots into responsive performers capable of reacting to audio.

Related papers

• RoboGesture: Real-Time Semantic-aligned Co-Speech Gestures Generation for Humanoid Interaction [33.04319504310729]
  RoboGesture is a robot-centric framework that co-designs data, modeling, and control to power a complete interactive human-humanoid system.<n>We first establish the RoboGesture dataset featuring over 300 gesture categories and develop an automated pipeline to synthesize large-scale audio-motion pairs.<n>Experiments on a Unitree G1 humanoid demonstrate that RoboGesture generates safer, more rhythmic, and more semantically appropriate responses compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-08-27T02:13:46Z)
• Semantic Audio-driven Understanding for Dynamic Humanoid Whole Body Control [0.0]
  We introduce a novel multi-modal orchestration framework for semantic audio-driven humanoid control.<n>The system processes continuous audio streams and routes them into music or speech branches.<n>Music input is handled via audio fingerprinting and semantic embeddings to retrieve track identity and temporal alignment.<n>Speech input is grounded into a discrete library of imitation-learned skills, enabling direct human-robot interaction.
  arXiv  Detail & Related papers  (2026-07-15T14:24:00Z)
• Co-policy: Responsive Human-Robot Co-Creation for Musical Performances [58.92814200971293]
  Co-policy is a framework for human-robot musical co-creation.<n>It separates semantic intent grounding, constrained musical variation, and visuomotor execution.<n>We show improved intent alignment, execution accuracy, and response frequency over diffusion-policy and ablated baselines.
  arXiv  Detail & Related papers  (2026-06-18T08:08:16Z)
• Bionic Human-Motion Style Transfer for Physically Executable Whole-Body Control of Humanoid Robots [13.21934742924018]
  This paper proposes a generation-to-control framework for exemplar-driven style transfer on humanoid robots.<n>A physics-aware latent diffusion model is developed to fuse style, content and trajectory conditions.<n>A preview-based whole-body tracking policy is trained with a cluster-and-distill strategy.
  arXiv  Detail & Related papers  (2026-06-02T11:59:01Z)
• EchoAvatar: Real-time Generative Avatar Animation from Audio Streams [31.328378976492775]
  We introduce a novel framework designed to generate continuous, coherent full-body motion from streaming speech and music with low latency.<n>By combining this controllability with stream audio-driven synthesis, our framework serves as a plug-and-play solution for transforming voice agents into interactive humanoid avatars.
  arXiv  Detail & Related papers  (2026-05-27T10:18:16Z)
• UniLS: End-to-End Audio-Driven Avatars for Unified Listening and Speaking [21.70536197968411]
  UniLS is the first end-to-end framework for generating unified speak-listen expressions, driven by only dual-track audio.<n>It delivers up to 44.1% improvement in listening metrics, generating significantly more diverse and natural listening expressions.
  arXiv  Detail & Related papers  (2025-12-10T05:25:58Z)
• From Language to Locomotion: Retargeting-free Humanoid Control via Motion Latent Guidance [55.31807046722006]
  Existing language-guided humanoid pipelines are cumbersome and untrustworthy.<n>We present RoboGhost, a language-free framework that conditions humanoid policies on language-grounded motion latents.<n>We show that RoboGhost substantially reduces deployment latency, improves success rates and tracking precision, and produces smooth, semantically aligned humanoids.
  arXiv  Detail & Related papers  (2025-10-16T17:57:47Z)
• MOSPA: Human Motion Generation Driven by Spatial Audio [83.31594478750682]
  We introduce the first comprehensive Spatial Audio-Driven Human Motion dataset, which contains diverse and high-quality spatial audio and motion data.<n>We develop a framework for human MOtion generation driven by SPatial Audio, termed MOSPA, which faithfully captures the relationship between body motion and spatial audio.<n>Our method achieves state-of-the-art performance on this task.
  arXiv  Detail & Related papers  (2025-07-16T06:33:11Z)
• Driving Animatronic Robot Facial Expression From Speech [7.8799497614708605]
  This paper introduces a novel, skinning-centric approach to drive animatronic robot facial expressions from speech input. The proposed approach employs linear skinning (LBS) as a unifying representation, guiding innovations in both embodiment design and motion synthesis. This approach demonstrates the capability to produce highly realistic facial expressions on an animatronic face in real-time at over 4000 fps on a single Nvidia GTX 4090.
  arXiv  Detail & Related papers  (2024-03-19T12:11:57Z)
• Audio-Driven Co-Speech Gesture Video Generation [92.15661971086746]
  We define and study this challenging problem of audio-driven co-speech gesture video generation. Our key insight is that the co-speech gestures can be decomposed into common motion patterns and subtle rhythmic dynamics. We propose a novel framework, Audio-driveN Gesture vIdeo gEneration (ANGIE), to effectively capture the reusable co-speech gesture patterns.
  arXiv  Detail & Related papers  (2022-12-05T15:28:22Z)
• Freeform Body Motion Generation from Speech [53.50388964591343]
  Body motion generation from speech is inherently difficult due to the non-deterministic mapping from speech to body motions. We introduce a novel freeform motion generation model (FreeMo) by equipping a two-stream architecture. Experiments demonstrate the superior performance against several baselines.
  arXiv  Detail & Related papers  (2022-03-04T13:03:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.