Evaluating and Preserving Lexical Stress in English-to-Chinese Speech-to-Speech Translation

• URL: http://arxiv.org/abs/2606.15266v1
• Date: Sat, 13 Jun 2026 11:57:59 GMT
• Title: Evaluating and Preserving Lexical Stress in English-to-Chinese Speech-to-Speech Translation
• Abstract summary: Cross-lingual transfer of lexical stress is a vital cue for emphasis and speaker intent.<n>We investigate English-to-Chinese S2ST stress transfer by constructing a stress-annotated Chinese dataset and an XLS-R-based Mandarin stress detector.
• Score: 12.104542664260139
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speech-to-speech translation (S2ST) systems have achieved impressive progress in semantic accuracy and speech naturalness. However, the cross-lingual transfer of lexical stress, a vital cue for emphasis and speaker intent, remains heavily underexplored, compounded by a lack of reliable automatic evaluation metrics for tonal languages like Chinese. We investigate English-to-Chinese S2ST stress transfer by constructing a stress-annotated Chinese dataset and an XLS-R-based Mandarin stress detector. Integrating this with the English EmphAssess system, we propose a novel objective metric for cross-lingual stress evaluation. Furthermore, we fine-tune CosyVoice3 to build a stress-aware S2ST system. Experiments demonstrate that our proposed S2ST architecture significantly outperforms existing systems in stress translation capability while maintaining competitive translation quality. Furthermore, our evaluation metric exhibits a strong correlation with human subjective judgments.

Related papers

• Disentangling Speaker and Language Effects in Cross-Lingual Speaker Verification for Iberian Languages [5.756864120253599]
  Cross-lingual speaker verification (SV) systems exhibit performance degradation when enrollment and test utterances are spoken in different languages.<n>We introduce a bilingual same-speaker evaluation set for five Iberian languages, enabling analysis of cross-lingual SV under constant speaker identity.
  arXiv  Detail & Related papers  (2026-07-01T16:44:19Z)
• Adapting Self-Supervised Speech Representations for Cross-lingual Dysarthria Detection in Parkinson's Disease [72.0406069194794]
  Speech representations often encode language-dependent structure that can confound dysarthria detection.<n>We propose a representation-level language shift that aligns source-language self-supervised speech representations with the target-language distribution.<n>We evaluate the approach on oral DDK recordings from Parkinson's disease speech datasets in Czech, German, and Spanish.
  arXiv  Detail & Related papers  (2026-03-23T17:23:39Z)
• Quantifying Cross-Lingual Transfer in Paralinguistic Speech Tasks [4.911970211082446]
  We introduce the Cross-Lingual Transfer Matrix (CLTM), a systematic method to quantify cross-lingual interactions between languages.<n>We apply the CLTM to two paralinguistic tasks, gender identification and speaker verification, using a multilingual HuBERT-based encoder.<n>Our results reveal distinct transfer patterns across tasks and languages, reflecting systematic, language-dependent effects.
  arXiv  Detail & Related papers  (2026-03-09T11:02:57Z)
• StressTransfer: Stress-Aware Speech-to-Speech Translation with Emphasis Preservation [10.037278049189073]
  We propose a stress-aware speech-to-speech translation (S2ST) system that preserves word-level emphasis.<n>Our method source-language stress into target-language tags that guide a controllable TTS model.
  arXiv  Detail & Related papers  (2025-10-15T06:32:24Z)
• A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning: Performance, Consistency, and Faithfulness Across Languages [48.68444770923683]
  We present the first comprehensive study of multilingual Chain-of-Thought (CoT) reasoning.<n>We measure language compliance, answer accuracy, and answer consistency when LRMs are prompt-hacked to think in a target language.<n>We find that the quality and effectiveness of thinking traces vary substantially depending on the prompt language.
  arXiv  Detail & Related papers  (2025-10-10T17:06:50Z)
• Evaluating Text Style Transfer: A Nine-Language Benchmark for Text Detoxification [66.69370876902222]
  We perform the first comprehensive multilingual study on evaluation of text detoxification system across nine languages.<n>We assess the effectiveness of modern neural-based evaluation models alongside prompting-based LLM-as-a-judge approaches.<n>Our findings provide a practical recipe for designing more reliable multilingual TST evaluation pipeline.
  arXiv  Detail & Related papers  (2025-07-21T12:38:07Z)
• An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios [76.11409260727459]
  This paper explores the language adaptation capability of ZMM-TTS, a recent SSL-based multilingual TTS system. We demonstrate that the similarity in phonetics between the pre-training and target languages, as well as the language category, affects the target language's adaptation performance.
  arXiv  Detail & Related papers  (2024-06-13T08:16:52Z)
• Attempt Towards Stress Transfer in Speech-to-Speech Machine Translation [0.0]
  Language diversity in India's education sector poses a significant challenge, hindering inclusivity. Despite democratization of knowledge through online educational content, the dominance of English limits accessibility. Despite existing Speech-to-Speech Machine Translation (SSMT) technologies, the lack of intonation in these systems gives monotonous translations. This paper introduces a dataset with stress annotations in Indian English and also a Text-to-Speech (TTS) architecture capable of incorporating stress into synthesized speech.
  arXiv  Detail & Related papers  (2024-03-07T03:21:19Z)
• Prosody in Cascade and Direct Speech-to-Text Translation: a case study on Korean Wh-Phrases [79.07111754406841]
  This work proposes using contrastive evaluation to measure the ability of direct S2TT systems to disambiguate utterances where prosody plays a crucial role. Our results clearly demonstrate the value of direct translation systems over cascade translation models.
  arXiv  Detail & Related papers  (2024-02-01T14:46:35Z)
• Lost in Interpreting: Speech Translation from Source or Interpreter? [0.0]
  We release 10 hours of recordings and transcripts of European Parliament speeches in English, with simultaneous interpreting into Czech and German. We evaluate quality and latency of speaker-based and interpreter-based spoken translation systems from English to Czech.
  arXiv  Detail & Related papers  (2021-06-17T09:32:49Z)
• Self-Attention with Cross-Lingual Position Representation [112.05807284056337]
  Position encoding (PE) is used to preserve the word order information for natural language processing tasks, generating fixed position indices for input sequences. Due to word order divergences in different languages, modeling the cross-lingual positional relationships might help SANs tackle this problem. We augment SANs with emphcross-lingual position representations to model the bilingually aware latent structure for the input sentence.
  arXiv  Detail & Related papers  (2020-04-28T05:23:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.