Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

• URL: http://arxiv.org/abs/2609.04250v1
• Date: Fri, 28 Aug 2026 21:02:39 GMT
• Title: Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue
• Abstract summary: Spoken dialogue models produce speech without motion, and co-speech motion models produce motion only from audio handed to them.<n>We present Motion- Omni, an end-to-end framework in which a spoken dialogue model outputs explicit facial expression together with hand, upper-body and lower-body motion.<n>We further release SwwenDA-500 and, to our knowledge, the first public evaluation protocol for open-ended full-body spoken dialogue.
• Score: 30.587755818246052
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: An avatar that holds a conversation should decide what to say and to move while saying it, yet these abilities live in separate model families: spoken dialogue models produce speech without motion, and co-speech motion models produce motion only from audio handed to them. The standard remedy is a cascade that first generates the spoken response and then runs a motion model over the finished audio, which requires a second full inference pass and precludes any joint optimisation between the two. We present Motion-Omni, an end-to-end framework in which a spoken dialogue model natively outputs explicit facial expression together with hand, upper-body and lower-body motion, generated directly from the hidden states that produce the speech. Joint training is not optional here: with the speech pathway frozen, motion remains misaligned with the audio, and co-adapting the LLM, Speech Generator and Motion Generator under both objectives is what recovers alignment while retaining spoken-dialogue ability. Supervision comes from a scalable, model-agnostic pipeline that pseudo-labels consistent-voice speech responses with a replaceable motion teacher, yielding 422,856 quality-ranked pairs (1,402 hours). We further release SwDA-500 and, to our knowledge, the first public evaluation protocol for stochastic open-ended full-body spoken dialogue, matching audio across motion systems while unifying rendering, automatic metrics, human evaluation, and latency measurement. Instantiated with a Qwen2.5-7B-Instruct backbone, Motion-Omni-Q7 matches the same-audio teacher cascade to within 2% on reference-free motion metrics while responding 5.4 x faster (RTF=0.78, faster than real time), surpasses all non-teacher cascades on beat correlation and diversity, and reaches a 2.62% word error rate, the lowest among the omni-modal systems compared.

Related papers

• Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models [49.372283081631366]
  AVSTE restores semantic speech tokens from noisy audio and lip video before they reach speech perception.<n>When integrated with Moshi, AVSTE improves average GPT-4ojudged response coherence from 1.42 1.91 under same-data-set speaker interference.
  arXiv  Detail & Related papers  (2026-09-08T07:59:13Z)
• Dual-Scale State-Space Modeling with Speaker-Wise Dynamic CRF for Speech Emotion Recognition in Conversation [1.6389263804357965]
  We propose DSSM-CRF, an audio-only architecture for emotion recognition.<n>Bidirectional state-space models encode fused self-supervised speech representations at frame and dialogue scales.<n> Consecutive utterances in a speaker's chain form a transition pair whose score combines a corpus-level transition matrix with a residual predicted from the two contextualized utterances.
  arXiv  Detail & Related papers  (2026-08-23T12:56:03Z)
• ReFree: Towards Realistic Co-Speech Video Generation via Reward-Free RL and Multilevel Speech Guidance [50.482989497576476]
  Speech-driven talking character animation seeks to generate life-like portrait videos that convey natural conversation behavior.<n>Existing approaches typically trade off precise phoneme-to-lip synchronization against dynamic facial expressions and head motion.<n>We propose ReFree-S2V, a flow-matching speech-to-portrait animation framework that builds upon a pretrained video generation model.
  arXiv  Detail & Related papers  (2026-06-11T13:00:58Z)
• Gesture2Speech: How Far Can Hand Movements Shape Expressive Speech? [11.96255700478909]
  We propose a novel multimodal text-to-speech framework, Gesture2Speech, that leverages visual gesture cues to modulate prosody in synthesized speech.<n>We show that Gesture2Speech outperforms state-of-the-art baselines in both speech naturalness and gesture-speech synchrony.<n>To the best of our knowledge, this is the first work to utilize hand gesture cues for prosody control in neural speech synthesis.
  arXiv  Detail & Related papers  (2026-03-20T10:17:10Z)
• MIBURI: Towards Expressive Interactive Gesture Synthesis [62.45332399212876]
  Embodied Conversational Agents (ECAs) aim to emulate human face-to-face interaction through speech, gestures, and facial expressions.<n>Existing solutions for ECAs produce rigid, low-diversity motions that are unsuitable for human-like interaction.<n>We present MIBURI, the first online, causal framework for generating expressive full-body gestures and facial expressions synchronized with real-time spoken dialogue.
  arXiv  Detail & Related papers  (2026-03-03T18:59:51Z)
• F-Actor: Controllable Conversational Behaviour in Full-Duplex Models [70.48189107402145]
  We present first open, instruction-following full-stage conversational speech model that can be trained efficiently under typical academic resource constraints.<n>Our model requires just 2,000 hours of data, without relying on large-scale pretraining or multi-stage pretraining.<n>Both the model and training code will be released to enable reproducible research on controllable full-like controllable full-stage speech systems.
  arXiv  Detail & Related papers  (2026-01-16T14:25:57Z)
• UniLS: End-to-End Audio-Driven Avatars for Unified Listening and Speaking [21.70536197968411]
  UniLS is the first end-to-end framework for generating unified speak-listen expressions, driven by only dual-track audio.<n>It delivers up to 44.1% improvement in listening metrics, generating significantly more diverse and natural listening expressions.
  arXiv  Detail & Related papers  (2025-12-10T05:25:58Z)
• HM-Talker: Hybrid Motion Modeling for High-Fidelity Talking Head Synthesis [90.74616208952791]
  HM-Talker is a novel framework for generating high-fidelity, temporally coherent talking heads.<n>Explicit cues use Action Units (AUs), anatomically defined facial muscle movements, alongside implicit features to minimize phoneme-viseme misalignment.
  arXiv  Detail & Related papers  (2025-08-14T12:01:52Z)
• It Takes Two: Real-time Co-Speech Two-person's Interaction Generation via Reactive Auto-regressive Diffusion Model [34.94330722832987]
  We introduce an audio-driven, auto-regressive system designed to synthesize dynamic movements for two characters during a conversation.<n>To the best of our knowledge, this is the first system capable of generating interactive full-body motions for two characters from speech in an online manner.
  arXiv  Detail & Related papers  (2024-12-03T12:31:44Z)
• Freeform Body Motion Generation from Speech [53.50388964591343]
  Body motion generation from speech is inherently difficult due to the non-deterministic mapping from speech to body motions. We introduce a novel freeform motion generation model (FreeMo) by equipping a two-stream architecture. Experiments demonstrate the superior performance against several baselines.
  arXiv  Detail & Related papers  (2022-03-04T13:03:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.