SiGnature: Explicit Motion Diffusion for Stylized Semantic Gesture

• URL: http://arxiv.org/abs/2606.15889v1
• Date: Sun, 14 Jun 2026 16:18:26 GMT
• Title: SiGnature: Explicit Motion Diffusion for Stylized Semantic Gesture
• Abstract summary: SiGnature is a framework for Stylized and Semantic Gesture generation.<n>It reconciles precise semantic control with high-fidelity style preservation.<n>It offers superior semantic motion control while maintaining smooth and natural co-speech gesture generation.
• Score: 31.872713283587604
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While recent advances in co-speech gesture generation have achieved impressive rhythmic synchronization, synthesizing gestures that are both semantically meaningful and faithful to a speaker's unique non-verbal style remains an open challenge. Semantic gestures, such as iconic shapes or deictic pointing, are statistically sparse, making them difficult to learn effectively within standard generative models. We present SiGnature, a framework for Stylized and Semantic Gesture generation that reconciles precise semantic control with high-fidelity style preservation. Unlike prevalent methods that rely on entangled latent representations, SiGnature operates in an explicit joint-rotation space. This design enables our core contribution, Joint Motion Integration (JMI), a training-free inference mechanism capable of injecting any external motion sequence, particularly in-the-wild semantic gestures, directly into the diffusion process. JMI automatically identifies the specific ``active joints'' conveying a semantic action and injects them into the generation, while relying on the diffusion backbone to synthesize the remaining body dynamics, including posture and flow, in accordance with the pre-learned style of the target speaker. This allows for the plug-and-play integration of arbitrary motions, including complex semantic gestures, without retraining or introducing the ``Frankenstein'' artifacts typical of cut-and-paste methods. Extensive experiments and perceptual studies demonstrate that SiGnature offers superior semantic motion control while maintaining smooth and natural co-speech gesture generation and preserving the distinct characteristics of the speaker, thereby outperforming state-of-the-art baselines.

Related papers

• Semantic Motion Anchors: Bridging Motion and Meaning in Co-Speech Gestures [58.6162288735234]
  We propose semantic motion anchors, natural-language abstractions of gesture motion capturing physical form and communicative intent.<n>Our method discretizes 3D gestures into body-hand motion primitives, verbalizes them into structured descriptions, and grounds them in the transcript.<n>On BEAT2, our method improves text-to-gesture R@1 by 8.2% over a direct text-motion baseline.
  arXiv  Detail & Related papers  (2026-05-28T22:01:11Z)
• HolisticSemGes: Semantic Grounding of Holistic Co-Speech Gesture Generation with Contrastive Flow-Matching [1.6916040234975798]
  We introduce a Contrastive Flow Matching-based co-speech gesture generation model.<n>It uses mismatched audio-text conditions as negatives, training the velocity field to follow the correct motion trajectory while repelling semantically incongruent trajectories.<n>Our model ensures cross-modal coherence by embedding text, audio, and holistic motion into a composite latent space via cosine and contrastive objectives.
  arXiv  Detail & Related papers  (2026-03-27T16:11:44Z)
• MIBURI: Towards Expressive Interactive Gesture Synthesis [62.45332399212876]
  Embodied Conversational Agents (ECAs) aim to emulate human face-to-face interaction through speech, gestures, and facial expressions.<n>Existing solutions for ECAs produce rigid, low-diversity motions that are unsuitable for human-like interaction.<n>We present MIBURI, the first online, causal framework for generating expressive full-body gestures and facial expressions synchronized with real-time spoken dialogue.
  arXiv  Detail & Related papers  (2026-03-03T18:59:51Z)
• 3DGesPolicy: Phoneme-Aware Holistic Co-Speech Gesture Generation Based on Action Control [3.606473077857744]
  3DGesPolicy is an action-based framework that reformulates holistic gesture generation as a continuous trajectory control problem.<n>By modeling frame-to-frame variations as unified holistic actions, our method effectively learns inter-frame holistic gesture motion patterns.<n>To further bridge the gap in expressive alignment, we propose a Gesture-Audio-Phoneme (GAP) fusion module.
  arXiv  Detail & Related papers  (2026-01-26T12:57:36Z)
• Sissi: Zero-shot Style-guided Image Synthesis via Semantic-style Integration [57.02757226679549]
  We introduce a training-free framework that reformulates style-guided synthesis as an in-context learning task.<n>We propose a Dynamic Semantic-Style Integration (DSSI) mechanism that reweights attention between semantic and style visual tokens.<n>Experiments show that our approach achieves high-fidelity stylization with superior semantic-style balance and visual quality.
  arXiv  Detail & Related papers  (2026-01-10T16:01:14Z)
• SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learning [0.6249768559720122]
  We propose a novel approach for semantic grounding in co-speech gesture generation.<n>Our approach starts with learning the motion prior through a vector-quantized variational autoencoder.<n>Our method outperforms state-of-the-art approaches across two benchmarks in co-speech gesture generation.
  arXiv  Detail & Related papers  (2025-07-25T15:10:15Z)
• Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesis [55.45253486141108]
  RAG-Gesture is a diffusion-based gesture generation approach to produce semantically rich gestures.<n>We achieve this by using explicit domain knowledge to retrieve motions from a database of co-speech gestures.<n>We propose a control paradigm for guidance, that allows the users to modulate the amount of influence each retrieval insertion has over the generated sequence.
  arXiv  Detail & Related papers  (2024-12-09T18:59:46Z)
• Emphasizing Semantic Consistency of Salient Posture for Speech-Driven Gesture Generation [44.78811546051805]
  Speech-driven gesture generation aims at synthesizing a gesture sequence synchronized with the input speech signal. Previous methods leverage neural networks to directly map a compact audio representation to the gesture sequence. We propose a novel speech-driven gesture generation method by emphasizing the semantic consistency of salient posture.
  arXiv  Detail & Related papers  (2024-10-17T17:22:59Z)
• Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis [25.822870767380685]
  We present Semantic Gesticulator, a framework designed to synthesize realistic gestures with strong semantic correspondence. Our system demonstrates robustness in generating gestures that are rhythmically coherent and semantically explicit. Our system outperforms state-of-the-art systems in terms of semantic appropriateness by a clear margin.
  arXiv  Detail & Related papers  (2024-05-16T05:09:01Z)
• Freeform Body Motion Generation from Speech [53.50388964591343]
  Body motion generation from speech is inherently difficult due to the non-deterministic mapping from speech to body motions. We introduce a novel freeform motion generation model (FreeMo) by equipping a two-stream architecture. Experiments demonstrate the superior performance against several baselines.
  arXiv  Detail & Related papers  (2022-03-04T13:03:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.