Semantic Motion Anchors: Bridging Motion and Meaning in Co-Speech Gestures

• URL: http://arxiv.org/abs/2605.30608v2
• Date: Mon, 01 Jun 2026 14:43:09 GMT
• Title: Semantic Motion Anchors: Bridging Motion and Meaning in Co-Speech Gestures
• Abstract summary: We propose semantic motion anchors, natural-language abstractions of gesture motion capturing physical form and communicative intent.<n>Our method discretizes 3D gestures into body-hand motion primitives, verbalizes them into structured descriptions, and grounds them in the transcript.<n>On BEAT2, our method improves text-to-gesture R@1 by 8.2% over a direct text-motion baseline.
• Score: 58.6162288735234
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning a shared representation between spoken text and gesture is central to co-speech gesture retrieval, synthesis, and understanding, but remains challenging for semantically meaningful gestures whose communicative intent is not captured by motion alone. Direct contrastive alignment between transcripts and continuous motion embeddings often overemphasizes low-level kinematics and misses the symbolic content of semantic gestures. We propose semantic motion anchors, natural-language abstractions of gesture motion capturing physical form and communicative intent. Our method discretizes 3D gestures into body-hand motion primitives, verbalizes them into structured descriptions, and grounds them in the transcript to provide auxiliary contrastive supervision. On BEAT2, our method improves text-to-gesture R@1 by 8.2% over a direct text-motion baseline and outperforms prior retrieval approaches on text to gesture and gesture to text retrieval directions. Beyond aggregate retrieval metrics, semantic motion anchor supervision helps retrieve gestures that are semantically meaningful for the spoken query, rather than defaulting to generic motion patterns. A downstream retrieval-augmented gesture generation study showed that users significantly preferred gestures retrieved by our approach over a retrieval-augmented generation baseline, demonstrating that semantically grounded retrieval translates to gestures that better convey communicative intent in downstream generation.

Related papers

• Recognizing Co-Speech Gestures in-the-Wild [52.804383193630166]
  Gesture Recognition in the Wild (GRW) dataset comprises 156,688 manually annotated video clips.<n>GRW spans a highly diverse 150-word taxonomy of physical actions, spatial descriptors, and abstract concepts.<n>We leverage GRW to train video models to classify gestures as semantic or not, (b) recognize the word corresponding to a co-speech gesture, and (c) temporally localize the gesture.
  arXiv  Detail & Related papers  (2026-05-29T17:55:17Z)
• CoordSpeaker: Exploiting Gesture Captioning for Coordinated Caption-Empowered Co-Speech Gesture Generation [44.84719308595376]
  CoordSpeaker is a comprehensive framework that enables coordinated caption-empowered co-speech gesture synthesis.<n>Our method produces high-quality gestures that are both rhythmically synchronized with speeches and semantically coherent with arbitrary captions.
  arXiv  Detail & Related papers  (2025-11-28T03:38:08Z)
• ImaGGen: Zero-Shot Generation of Co-Speech Semantic Gestures Grounded in Language and Image Input [0.0]
  This paper tackles a core challenge in co-speech gesture synthesis: generating iconic or deictic gestures that are semantically coherent with a verbal utterance.<n>We introduce a zero-shot system that generates gestures from a given language input and additionally is informed by imagistic input, without manual annotation or human intervention.<n>Our results highlight the importance of context-aware semantic gestures for creating expressive and collaborative virtual agents or avatars.
  arXiv  Detail & Related papers  (2025-10-20T15:01:56Z)
• Intentional Gesture: Deliver Your Intentions with Gestures for Speech [47.34315450130868]
  Intentional-Gesture is a novel framework that casts gesture generation as an intention-reasoning task grounded in high-level communicative functions.<n>Our framework offers a modular foundation for expressive gesture generation in digital humans and embodied AI.
  arXiv  Detail & Related papers  (2025-05-21T07:24:51Z)
• Understanding Co-speech Gestures in-the-wild [52.5993021523165]
  We introduce a new framework for co-speech gesture understanding in the wild.<n>We propose three new tasks and benchmarks to evaluate a model's capability to comprehend gesture-speech-text associations.<n>We present a new approach that learns a tri-modal video-gesture-speech-text representation to solve these tasks.
  arXiv  Detail & Related papers  (2025-03-28T17:55:52Z)
• Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesis [55.45253486141108]
  RAG-Gesture is a diffusion-based gesture generation approach to produce semantically rich gestures.<n>We achieve this by using explicit domain knowledge to retrieve motions from a database of co-speech gestures.<n>We propose a control paradigm for guidance, that allows the users to modulate the amount of influence each retrieval insertion has over the generated sequence.
  arXiv  Detail & Related papers  (2024-12-09T18:59:46Z)
• Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis [25.822870767380685]
  We present Semantic Gesticulator, a framework designed to synthesize realistic gestures with strong semantic correspondence. Our system demonstrates robustness in generating gestures that are rhythmically coherent and semantically explicit. Our system outperforms state-of-the-art systems in terms of semantic appropriateness by a clear margin.
  arXiv  Detail & Related papers  (2024-05-16T05:09:01Z)
• ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis [50.69464138626748]
  We present ConvoFusion, a diffusion-based approach for multi-modal gesture synthesis. Our method proposes two guidance objectives that allow the users to modulate the impact of different conditioning modalities. Our method is versatile in that it can be trained either for generating monologue gestures or even the conversational gestures.
  arXiv  Detail & Related papers  (2024-03-26T17:59:52Z)
• LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generation [41.42316077949012]
  We introduce LivelySpeaker, a framework that realizes semantics-aware co-speech gesture generation. Our method decouples the task into two stages: script-based gesture generation and audio-guided rhythm refinement. Our novel two-stage generation framework also enables several applications, such as changing the gesticulation style.
  arXiv  Detail & Related papers  (2023-09-17T15:06:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.