GesGPT: Speech Gesture Synthesis With Text Parsing from ChatGPT

• URL: http://arxiv.org/abs/2303.13013v3
• Date: Tue, 28 May 2024 02:53:44 GMT
• Title: GesGPT: Speech Gesture Synthesis With Text Parsing from ChatGPT
• Authors: Nan Gao, Zeyu Zhao, Zhi Zeng, Shuwu Zhang, Dongdong Weng, Yihua Bao,
• Abstract summary: Gesture synthesis has gained significant attention as a critical research field. Deep learning-based approaches have achieved remarkable progress, but they often overlook the rich semantic information present in the text. We propose GesGPT, a novel approach to gesture generation that leverages the semantic analysis capabilities of large language models.
• Score: 8.18076191897917
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Gesture synthesis has gained significant attention as a critical research field, aiming to produce contextually appropriate and natural gestures corresponding to speech or textual input. Although deep learning-based approaches have achieved remarkable progress, they often overlook the rich semantic information present in the text, leading to less expressive and meaningful gestures. In this letter, we propose GesGPT, a novel approach to gesture generation that leverages the semantic analysis capabilities of large language models , such as ChatGPT. By capitalizing on the strengths of LLMs for text analysis, we adopt a controlled approach to generate and integrate professional gestures and base gestures through a text parsing script, resulting in diverse and meaningful gestures. Firstly, our approach involves the development of prompt principles that transform gesture generation into an intention classification problem using ChatGPT. We also conduct further analysis on emphasis words and semantic words to aid in gesture generation. Subsequently, we construct a specialized gesture lexicon with multiple semantic annotations, decoupling the synthesis of gestures into professional gestures and base gestures. Finally, we merge the professional gestures with base gestures. Experimental results demonstrate that GesGPT effectively generates contextually appropriate and expressive gestures.

Related papers

• Emphasizing Semantic Consistency of Salient Posture for Speech-Driven Gesture Generation [44.78811546051805]
  Speech-driven gesture generation aims at synthesizing a gesture sequence synchronized with the input speech signal. Previous methods leverage neural networks to directly map a compact audio representation to the gesture sequence. We propose a novel speech-driven gesture generation method by emphasizing the semantic consistency of salient posture.
  arXiv  Detail & Related papers  (2024-10-17T17:22:59Z)
• ExpGest: Expressive Speaker Generation Using Diffusion Model and Hybrid Audio-Text Guidance [11.207513771079705]
  We introduce ExpGest, a novel framework leveraging synchronized text and audio information to generate expressive full-body gestures. Unlike AdaIN or one-hot encoding methods, we design a noise emotion classifier for optimizing adversarial direction noise. We show that ExpGest achieves more expressive, natural, and controllable global motion in speakers compared to state-of-the-art models.
  arXiv  Detail & Related papers  (2024-10-12T07:01:17Z)
• Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis [25.822870767380685]
  We present Semantic Gesticulator, a framework designed to synthesize realistic gestures with strong semantic correspondence. Our system demonstrates robustness in generating gestures that are rhythmically coherent and semantically explicit. Our system outperforms state-of-the-art systems in terms of semantic appropriateness by a clear margin.
  arXiv  Detail & Related papers  (2024-05-16T05:09:01Z)
• Dynamic Typography: Bringing Text to Life via Video Diffusion Prior [73.72522617586593]
  We present an automated text animation scheme, termed "Dynamic Typography" It deforms letters to convey semantic meaning and infuses them with vibrant movements based on user prompts. Our technique harnesses vector graphics representations and an end-to-end optimization-based framework.
  arXiv  Detail & Related papers  (2024-04-17T17:59:55Z)
• ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis [50.69464138626748]
  We present ConvoFusion, a diffusion-based approach for multi-modal gesture synthesis. Our method proposes two guidance objectives that allow the users to modulate the impact of different conditioning modalities. Our method is versatile in that it can be trained either for generating monologue gestures or even the conversational gestures.
  arXiv  Detail & Related papers  (2024-03-26T17:59:52Z)
• LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generation [41.42316077949012]
  We introduce LivelySpeaker, a framework that realizes semantics-aware co-speech gesture generation. Our method decouples the task into two stages: script-based gesture generation and audio-guided rhythm refinement. Our novel two-stage generation framework also enables several applications, such as changing the gesticulation style.
  arXiv  Detail & Related papers  (2023-09-17T15:06:11Z)
• QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation [8.604430209445695]
  Speech-driven gesture generation is highly challenging due to the random jitters of human motion. We introduce a novel quantization-based and phase-guided motion-matching framework. Our method outperforms recent approaches on speech-driven gesture generation.
  arXiv  Detail & Related papers  (2023-05-18T16:31:25Z)
• Audio-Driven Co-Speech Gesture Video Generation [92.15661971086746]
  We define and study this challenging problem of audio-driven co-speech gesture video generation. Our key insight is that the co-speech gestures can be decomposed into common motion patterns and subtle rhythmic dynamics. We propose a novel framework, Audio-driveN Gesture vIdeo gEneration (ANGIE), to effectively capture the reusable co-speech gesture patterns.
  arXiv  Detail & Related papers  (2022-12-05T15:28:22Z)
• Vision-Language Pre-Training for Boosting Scene Text Detectors [57.08046351495244]
  We specifically adapt vision-language joint learning for scene text detection. We propose to learn contextualized, joint representations through vision-language pre-training. The pre-trained model is able to produce more informative representations with richer semantics.
  arXiv  Detail & Related papers  (2022-04-29T03:53:54Z)
• Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation [107.10239561664496]
  We propose a novel framework named Hierarchical Audio-to-Gesture (HA2G) for co-speech gesture generation. The proposed method renders realistic co-speech gestures and outperforms previous methods in a clear margin.
  arXiv  Detail & Related papers  (2022-03-24T16:33:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.