Ouvia: A User-centered Framework for Measuring Usability of Speech Translation in Real-World Communication Scenarios

• URL: http://arxiv.org/abs/2606.06177v1
• Date: Thu, 04 Jun 2026 13:52:21 GMT
• Title: Ouvia: A User-centered Framework for Measuring Usability of Speech Translation in Real-World Communication Scenarios
• Abstract summary: Speech translation (ST) is increasingly adopted in user applications, yet its evaluation focuses on decontextualized testbeds and holistic quality.<n>We introduce Ouvia, an evaluation framework for measuring user-perceived usability of speech translation outputs in real-world settings.
• Score: 63.167237308464536
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Speech translation (ST) is increasingly adopted in user applications, yet its evaluation largely focuses on decontextualized testbeds and holistic quality, rather than end users' communication needs. We introduce Ouvia, an evaluation framework for measuring user-perceived usability of speech translation outputs in real-world settings. Ouvia focuses on one-to-one communication: an English speaker needs to convey a request to a Portuguese speaker, and the message is automatically translated. Through a custom web app and multi-phase study design, we collect more than 1,750 such interactions in healthcare and everyday situations, mediated by four ST systems, involving speakers from three English dialects and two genders. We find that modern ST serves people only to a limited extent -- only around half of interactions are rated as usable -- with significant gaps in reported usability across demographic groups. Moreover, among quality metrics, we find that QA-based evaluation is a substantially stronger predictor of real-world usability than standard approaches. Together, these findings stress the importance of situated, user-centered evaluation frameworks that go beyond holistic quality scores and attend to who the technology serves -- and how well.

Related papers

• Domain-Specific Evaluation of Text-to-Speech Systems: A Multi-Metric Benchmarking Study [1.224954637705144]
  This paper presents a reproducible, multi-metric benchmarking framework for systematic evaluation of modern TTS systems.<n>Four state-of-the-art TTS systems -- Indic-Parler-TTS, MMS-TTS, Microsoft Edge TTS, and Google Gemini TTS -- are evaluated.<n>Results reveal substantial variation in TTS performance across speech domains, with emotional speech consistently presenting the greatest synthesis challenge.
  arXiv  Detail & Related papers  (2026-08-03T13:49:11Z)
• Bridging the Usability Gap: Lessons from Interpreting Studies for Machine Interpreting Design [0.0]
  Machine interpreting (MI) is the live, real-time application of speech translation.<n>Yet the user experience remains far inferior to interpreter-mediated communication.<n>This paper defines MI as a distinct subfield of speech translation.
  arXiv  Detail & Related papers  (2026-06-14T20:41:59Z)
• MUSCAT: MUltilingual, SCientific ConversATion Benchmark [60.87925076316812]
  The goal of multilingual speech technology is to facilitate seamless communication between individuals speaking different languages.<n>To create this experience, speech technology needs to address several challenges: Handling mixed multilingual input, specific vocabulary, and code-switching.<n>We propose a new benchmark to evaluate current Automatic Speech Recognition (ASR) systems, whether they are able to handle these challenges.
  arXiv  Detail & Related papers  (2026-04-17T10:39:01Z)
• Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition [52.63676763985825]
  We propose a semantic-aware evaluation metric to assess recognition quality beyond token-level accuracy.<n>We design an agent framework to simulate human-like multi-turn interaction, enabling iterative refinement of recognition outputs.<n>We will release the code to facilitate future research in interactive and agentic ASR.
  arXiv  Detail & Related papers  (2026-04-10T09:02:42Z)
• Multilingual Self-Taught Faithfulness Evaluators [11.200203292660758]
  Self-Taught Evaluators for Multilingual Faithfulness is a framework that learns exclusively from synthetic multilingual summarization data.<n>Our framework shows improvements over existing baselines, including state-of-the-art English evaluators and machine translation-based approaches.
  arXiv  Detail & Related papers  (2025-07-28T12:01:59Z)
• WildSpeech-Bench: Benchmarking End-to-End SpeechLLMs in the Wild [40.210224623581155]
  Multi-modal Large Language Models (LLMs) have demonstrated strong capabilities of direct speech interaction.<n>Existing evaluation methods often adapt text-based benchmarks, overlooking speech's unique characteristics and challenges.<n>Here, we introduce the first comprehensive benchmark designed to systematically evaluate end-to-end speechLLMs in practical speech conversations.
  arXiv  Detail & Related papers  (2025-06-27T03:18:45Z)
• CLAIR-A: Leveraging Large Language Models to Judge Audio Captions [73.51087998971418]
  evaluating machine-generated audio captions is a complex task that requires considering diverse factors.<n>We propose CLAIR-A, a simple and flexible method that leverages the zero-shot capabilities of large language models.<n>In our evaluations, CLAIR-A better predicts human judgements of quality compared to traditional metrics.
  arXiv  Detail & Related papers  (2024-09-19T17:59:52Z)
• Is Context Helpful for Chat Translation Evaluation? [23.440392979857247]
  We conduct a meta-evaluation of existing sentence-level automatic metrics to assess the quality of machine-translated chats. We find that reference-free metrics lag behind reference-based ones, especially when evaluating translation quality in out-of-English settings. We propose a new evaluation metric, Context-MQM, that utilizes bilingual context with a large language model.
  arXiv  Detail & Related papers  (2024-03-13T07:49:50Z)
• The Conversational Short-phrase Speaker Diarization (CSSD) Task: Dataset, Evaluation Metric and Baselines [63.86406909879314]
  This paper describes the Conversational Short-phrases Speaker Diarization (CSSD) task. It consists of training and testing datasets, evaluation metric and baselines. In the metric aspect, we design the new conversational DER (CDER) evaluation metric, which calculates the SD accuracy at the utterance level.
  arXiv  Detail & Related papers  (2022-08-17T03:26:23Z)
• Towards the evaluation of simultaneous speech translation from a communicative perspective [0.0]
  We present the results of an experiment aimed at evaluating the quality of a simultaneous speech translation engine. We found better performance for the human interpreters in terms of intelligibility, while the machine performs slightly better in terms of informativeness.
  arXiv  Detail & Related papers  (2021-03-15T13:09:00Z)
• Learning an Unreferenced Metric for Online Dialogue Evaluation [53.38078951628143]
  We propose an unreferenced automated evaluation metric that uses large pre-trained language models to extract latent representations of utterances. We show that our model achieves higher correlation with human annotations in an online setting, while not requiring true responses for comparison during inference.
  arXiv  Detail & Related papers  (2020-05-01T20:01:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.