DuoGesture: Neuro-Inspired and Biomechanically Informed Dual-Stream Co-Speech Gesture Generation

• URL: http://arxiv.org/abs/2605.26236v1
• Date: Mon, 25 May 2026 18:03:41 GMT
• Title: DuoGesture: Neuro-Inspired and Biomechanically Informed Dual-Stream Co-Speech Gesture Generation
• Abstract summary: Existing holistic gesture models mix lexically grounded semantic gestures with frequent prosody-aligned beat gestures.<n>We propose emphDuoGesture, a neuro-inspired and biomechanically informed dual-stream approach that decomposes co-speech gesture synthesis into coupled semantic and beat streams.
• Score: 27.296930205552954
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Co-speech gesture generation requires both semantic expressivity and biomechanically plausible rhythmic motion. Existing holistic gesture models mix lexically grounded semantic gestures with frequent prosody-aligned beat gestures. This limits semantic grounding, speech-motion alignment, and kinematic smoothness. We propose \emph{DuoGesture}, a neuro-inspired and biomechanically informed dual-stream approach that decomposes co-speech gesture synthesis into coupled semantic and beat streams. The two streams are coordinated by a \emph{Semantic Variational Information Bottleneck}, a stochastic frame-level gate that learns when semantic gestures should override rhythmic beat motion. The semantic stream is controlled by \emph{Motion-Grounded Semantic Conditioning}, which replaces purely linguistic word embeddings with motion-language representations to provide motion-aligned semantic priors for long-tailed lexical triggers of gestures. The beat stream is further regularised by an \emph{Inertial Beat Prior}, an anthropometry-weighted arm-chain module that reduces jitter and improves rhythmic consistency without constraining semantic frames. Objective evaluations and subjective experiments show that DuoGesture outperforms strong holistic baselines, while component ablations confirm the complementary roles of semantic grounding, stochastic stream selection, and biomechanical regularisation.

Related papers

• SiGnature: Explicit Motion Diffusion for Stylized Semantic Gesture [31.872713283587604]
  SiGnature is a framework for Stylized and Semantic Gesture generation.<n>It reconciles precise semantic control with high-fidelity style preservation.<n>It offers superior semantic motion control while maintaining smooth and natural co-speech gesture generation.
  arXiv  Detail & Related papers  (2026-06-14T16:18:26Z)
• Recognizing Co-Speech Gestures in-the-Wild [52.804383193630166]
  Gesture Recognition in the Wild (GRW) dataset comprises 156,688 manually annotated video clips.<n>GRW spans a highly diverse 150-word taxonomy of physical actions, spatial descriptors, and abstract concepts.<n>We leverage GRW to train video models to classify gestures as semantic or not, (b) recognize the word corresponding to a co-speech gesture, and (c) temporally localize the gesture.
  arXiv  Detail & Related papers  (2026-05-29T17:55:17Z)
• Semantic Motion Anchors: Bridging Motion and Meaning in Co-Speech Gestures [58.6162288735234]
  We propose semantic motion anchors, natural-language abstractions of gesture motion capturing physical form and communicative intent.<n>Our method discretizes 3D gestures into body-hand motion primitives, verbalizes them into structured descriptions, and grounds them in the transcript.<n>On BEAT2, our method improves text-to-gesture R@1 by 8.2% over a direct text-motion baseline.
  arXiv  Detail & Related papers  (2026-05-28T22:01:11Z)
• Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation [50.411841997631484]
  We present Unison, a unified framework that promotes coherence across the motion, speech, and sound modalities.<n>We show that Unison achieves state-of-the-art performance in both audio perceptual quality and cross-modal synchronization.
  arXiv  Detail & Related papers  (2026-05-09T06:32:54Z)
• MIBURI: Towards Expressive Interactive Gesture Synthesis [62.45332399212876]
  Embodied Conversational Agents (ECAs) aim to emulate human face-to-face interaction through speech, gestures, and facial expressions.<n>Existing solutions for ECAs produce rigid, low-diversity motions that are unsuitable for human-like interaction.<n>We present MIBURI, the first online, causal framework for generating expressive full-body gestures and facial expressions synchronized with real-time spoken dialogue.
  arXiv  Detail & Related papers  (2026-03-03T18:59:51Z)
• Beyond Global Alignment: Fine-Grained Motion-Language Retrieval via Pyramidal Shapley-Taylor Learning [56.6025512458557]
  Motion-language retrieval aims to bridge the semantic gap between natural language and human motion.<n>Existing approaches predominantly focus on aligning entire motion sequences with global textual representations.<n>We propose a novel Pyramidal Shapley-Taylor (PST) learning framework for fine-grained motion-language retrieval.
  arXiv  Detail & Related papers  (2026-01-29T16:00:12Z)
• SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasis [19.764460501254607]
  A good co-speech motion generation cannot be achieved without a careful integration of common rhythmic motion and rare yet essential semantic motion.<n>We propose SemTalk for holistic co-speech motion generation with frame-level semantic emphasis.
  arXiv  Detail & Related papers  (2024-12-21T10:16:07Z)
• Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings [27.352570417976153]
  We present a novel co-speech gesture synthesis method that achieves convincing results both on the rhythm and semantics. For the rhythm, our system contains a robust rhythm-based segmentation pipeline to ensure the temporal coherence between the vocalization and gestures explicitly. For the gesture semantics, we devise a mechanism to effectively disentangle both low- and high-level neural embeddings of speech and motion based on linguistic theory.
  arXiv  Detail & Related papers  (2022-10-04T08:19:06Z)
• Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation [107.10239561664496]
  We propose a novel framework named Hierarchical Audio-to-Gesture (HA2G) for co-speech gesture generation. The proposed method renders realistic co-speech gestures and outperforms previous methods in a clear margin.
  arXiv  Detail & Related papers  (2022-03-24T16:33:29Z)
• Freeform Body Motion Generation from Speech [53.50388964591343]
  Body motion generation from speech is inherently difficult due to the non-deterministic mapping from speech to body motions. We introduce a novel freeform motion generation model (FreeMo) by equipping a two-stream architecture. Experiments demonstrate the superior performance against several baselines.
  arXiv  Detail & Related papers  (2022-03-04T13:03:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.