Co-Speech with You: Training-Free Personalization of Robot Co-Speech Gestures

• URL: http://arxiv.org/abs/2609.13876v1
• Date: Sat, 12 Sep 2026 11:02:51 GMT
• Title: Co-Speech with You: Training-Free Personalization of Robot Co-Speech Gestures
• Abstract summary: Personal robots should adapt their co-speech gesture style to a new user without requiring model retraining.<n>We present a training-free personalization pipeline that combines a frozen audio-conditioned diffusion prior with a gesture style encoder.<n>We evaluate the system on held-out speakers using Style Recognition Accuracy (SRA) and Frechet Gesture Distance (FGD) to measure personalization and motion quality.
• Score: 0.8555238739527985
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Personal robots should adapt their co-speech gesture style to a new user without requiring model retraining. We present a training-free personalization pipeline that combines a frozen audio-conditioned diffusion prior with a gesture style encoder and lightweight conditioning adapters. The encoder is first trained to discriminate speaker identities and then jointly refined with the adapters using the diffusion objective, enabling a reusable style embedding to be extracted from approximately 10 seconds of enrollment motion through a single forward pass. To support this setting, we also release a Quest~3 capture application and a dataset of spontaneous co-speech motion from ten participants. We evaluate the system on held-out speakers using Style Recognition Accuracy (SRA) and Fr'echet Gesture Distance (FGD) to measure personalization and motion quality. Our approach improves SRA from 27.6\% for the frozen prior to 69.5\% while preserving motion quality (FGD 34.2 versus 34.8), and replacing the enrollment embedding with another person's reduces SRA to 11.4\%. The generated gestures are retargeted to a physical NAO robot, and this improvement also transfers to the robot deployment setting, where speech is synthesized using five TTS voices, retaining 67.3\% SRA.

Related papers

• Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue [30.587755818246052]
  Spoken dialogue models produce speech without motion, and co-speech motion models produce motion only from audio handed to them.<n>We present Motion- Omni, an end-to-end framework in which a spoken dialogue model outputs explicit facial expression together with hand, upper-body and lower-body motion.<n>We further release SwwenDA-500 and, to our knowledge, the first public evaluation protocol for open-ended full-body spoken dialogue.
  arXiv  Detail & Related papers  (2026-08-28T21:02:39Z)
• The Talking Robot: Distortion-Robust Acoustic Models for Robot-Robot Communication [6.356219992638542]
  Artoo is a learned acoustic communication system for robots.<n>It replaces hand-designed signal processing with end-to-end co-trained neural networks.
  arXiv  Detail & Related papers  (2026-03-07T07:06:41Z)
• Human-Centered Editable Speech-to-Sign-Language Generation via Streaming Conformer-Transformer and Resampling Hook [0.0]
  Existing end-to-end sign-language animation systems suffer from low naturalness, limited facial/body expressivity, and no user control.<n>We propose a human-centered, real-time speech-to-sign animation framework.<n> Deployed on Unity3D, our system achieves a 13 ms frame-in time and a 103 ms latency.
  arXiv  Detail & Related papers  (2025-06-17T16:08:48Z)
• Homogeneous Speaker Features for On-the-Fly Dysarthric and Elderly Speaker Adaptation [71.31331402404662]
  This paper proposes two novel data-efficient methods to learn dysarthric and elderly speaker-level features. Speaker-regularized spectral basis embedding-SBE features that exploit a special regularization term to enforce homogeneity of speaker features in adaptation. Feature-based learning hidden unit contributions (f-LHUC) that are conditioned on VR-LH features that are shown to be insensitive to speaker-level data quantity in testtime adaptation.
  arXiv  Detail & Related papers  (2024-07-08T18:20:24Z)
• Towards Evaluating the Robustness of Automatic Speech Recognition Systems via Audio Style Transfer [8.948537516293328]
  We propose an attack on Automatic Speech Recognition (ASR) systems based on user-customized style transfer. Our method can meet the need for user-customized styles and achieve a success rate of 82% in attacks.
  arXiv  Detail & Related papers  (2024-05-15T16:05:24Z)
• From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition [50.93943755401025]
  We propose a new parameter-efficient learning framework based on neural model reprogramming for cross-lingual speech recognition. We design different auxiliary neural architectures focusing on learnable pre-trained feature enhancement. Our methods outperform existing ASR tuning architectures and their extension with self-supervised losses.
  arXiv  Detail & Related papers  (2023-01-19T02:37:56Z)
• Exploiting Cross-domain And Cross-Lingual Ultrasound Tongue Imaging Features For Elderly And Dysarthric Speech Recognition [55.25565305101314]
  Articulatory features are invariant to acoustic signal distortion and have been successfully incorporated into automatic speech recognition systems. This paper presents a cross-domain and cross-lingual A2A inversion approach that utilizes the parallel audio and ultrasound tongue imaging (UTI) data of the 24-hour TaL corpus in A2A model pre-training. Experiments conducted on three tasks suggested incorporating the generated articulatory features consistently outperformed the baseline TDNN and Conformer ASR systems.
  arXiv  Detail & Related papers  (2022-06-15T07:20:28Z)
• Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learning [52.73083137245969]
  We present a generative adversarial network to synthesize 3D pose sequences of co-speech upper-body gestures with appropriate affective expressions. Our network consists of two components: a generator to synthesize gestures from a joint embedding space of features encoded from the input speech and the seed poses, and a discriminator to distinguish between the synthesized pose sequences and real 3D pose sequences.
  arXiv  Detail & Related papers  (2021-07-31T15:13:39Z)
• Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformers [56.56220390953412]
  We extend our prior work by introducing the Conformer architecture to further improve the accuracy. We demonstrate that the extended Transformer provides state-of-the-art end-to-end ASR performance.
  arXiv  Detail & Related papers  (2021-04-19T16:18:00Z)
• A Method to Reveal Speaker Identity in Distributed ASR Training, and How to Counter It [3.18475216176047]
  We design the first method for revealing the identity of the speaker of a training utterance with access only to a gradient. We show that it is possible to reveal the speaker's identity with 34% top-1 accuracy (51% top-5 accuracy) on the LibriSpeech dataset.
  arXiv  Detail & Related papers  (2021-04-15T23:15:12Z)
• Dynamic Acoustic Unit Augmentation With BPE-Dropout for Low-Resource End-to-End Speech Recognition [62.94773371761236]
  We consider building an effective end-to-end ASR system in low-resource setups with a high OOV rate. We propose a method of dynamic acoustic unit augmentation based on the BPE-dropout technique. Our monolingual Turkish Conformer established a competitive result with 22.2% character error rate (CER) and 38.9% word error rate (WER)
  arXiv  Detail & Related papers  (2021-03-12T10:10:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.