Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does

• URL: http://arxiv.org/abs/2608.19515v2
• Date: Fri, 21 Aug 2026 00:53:33 GMT
• Title: Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does
• Abstract summary: We introduce Hear2Act, a unified evaluation protocol for text and spoken assistants.<n>We evaluate decisions under transcript, audio, and concern-state access.<n>We show that prosody matters when lexical evidence is insufficient.
• Score: 27.909684871924004
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prosodic cues can convey task-relevant information that alters the trajectory and outcome of a task-oriented dialogue, even when the words themselves remain unchanged. Yet existing benchmarks typically evaluate prosodic perception, response appropriateness, and task-oriented dialogue in isolation, making it difficult to test whether prosodic evidence changes downstream decisions. We introduce Hear2Act, a unified evaluation protocol for text and spoken assistants with 480 persona-grounded scenarios, hidden user concerns, and objectively verifiable outcomes. For each scenario, we keep the task and user needs fixed while varying whether the same concern is conveyed explicitly in words or primarily through prosody, and evaluate decisions under transcript, audio, and concern-state access. Using Hear2Act, we evaluate two audio-capable LLMs. Under Prosody-mediated feedback, adding audio to the transcript changes the average optimal-solution rate only from 14.6% to 15.3%. In contrast, when models infer the concern status from audio, represent it in text, and use it for next-action selection, the rate rises to 39.6%, close to 40.7% with the ground-truth state. This contrast, however, largely disappears under Explicit lexical feedback, where the concern is verbally mentioned in the utterance. Together, these results show that prosody matters when lexical evidence is insufficient, and that audio-capable LLMs can recover information from speech but do not reliably carry it into action without an explicit intermediate representation.

Related papers

• DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents [67.07361089429757]
  We measure real-time floor adherence using six real-time speech systems.<n>We find architecture-dependent trade-offs.<n>GPT-Realtime, Mini-o Duplex, Fun-Audio-Chat strongly adhere to persona-consistent content.
  arXiv  Detail & Related papers  (2026-09-03T06:28:02Z)
• When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue [18.262137202372454]
  We formalize failure mode as cross-modal disagreement, where transcripts suggest plausible but incorrect surface interpretations.<n>We develop a scalable framework that identifies text-biased surface interpretations and converts disagreement regions into conflict QA examples.<n>We further develop an agentic-style reasoning framework that converts speech into an Audio Twin, a text-readable representation of localized acoustic cues.
  arXiv  Detail & Related papers  (2026-08-27T14:26:46Z)
• Do Audio Language Models Use Paralinguistic Evidence? Counterfactual Audits for Response Evaluation [21.291909657115607]
  We introduce counterfactual audits for paralinguistic response evaluation.<n>Each audit item holds the transcript fixed while varying affect, prosody, or the timing of an affective shift.<n>This yields useful diagnostic states that identify different sources of judge failures.
  arXiv  Detail & Related papers  (2026-08-07T02:24:12Z)
• Experience-Calibrated Contrastive Decoding for Mitigating Hallucinations in LM-Based Text-to-Speech [25.75512357205199]
  Language model-based text-to-speech (LM-based TTS) remains vulnerable to speech hallucinations that deviate from the target text.<n>We present a conditional information view that distinguishes text-derived alignment information from experience information.<n>We propose Experience-Calibrated Contrastive Decoding (ECCD), a training-free method that strengthens alignment support while preserving useful experience information.
  arXiv  Detail & Related papers  (2026-08-01T15:41:55Z)
• Afrispeech Semantics: Evaluating Audio Semantic Reasoning in Spoken Language Models Across Domains and Accents [3.6734440150955368]
  Audio language models (ALMs) are increasingly used for speech-based understanding.<n>The effects of accent variation, domain shift, and semantic over-inference on audio reasoning are poorly understood.<n>We evaluate audio language models across five semantic and paralinguistic reasoning tasks.
  arXiv  Detail & Related papers  (2026-05-11T20:27:40Z)
• From Fact to Judgment: Investigating the Impact of Task Framing on LLM Conviction in Dialogue Systems [8.8953040142657]
  We investigate how an LLM's conviction is changed when a task is reframed from a direct factual query to a Conversational Judgment Task.<n>We apply pressure in the form of a simple rebuttal ("The previous answer is incorrect.") to both conditions.<n>Our findings show that while some models like GPT-4o-mini reveal sycophantic tendencies under social framing tasks, others like Llama-8B-Instruct become overly-critical.
  arXiv  Detail & Related papers  (2025-11-14T00:55:28Z)
• AHELM: A Holistic Evaluation of Audio-Language Models [78.20477815156484]
  multimodal audio-language models (ALMs) take interleaved audio and text as input and output text.<n>AHELM is a benchmark that aggregates various datasets -- including 2 new synthetic audio-text datasets called PARADE and CoRe-Bench.<n>We also standardize the prompts, inference parameters, and evaluation metrics to ensure equitable comparisons across models.
  arXiv  Detail & Related papers  (2025-08-29T07:40:39Z)
• Audio Large Language Models Can Be Descriptive Speech Quality Evaluators [46.765203628127345]
  We introduce the first natural language-based speech evaluation corpus, generated from authentic human ratings.<n>This corpus offers detailed analysis across multiple dimensions and identifies causes of quality degradation.<n>We propose an alignment approach with LLM distillation (ALLD) to guide the audio LLM in extracting relevant information from raw speech.
  arXiv  Detail & Related papers  (2025-01-27T22:47:51Z)
• Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models [58.43486430996411]
  Large Audio-Language Models (LALMs) have recently unlocked audio dialogue capabilities, enabling direct spoken exchanges with humans.<n>We propose an Audio Dialogue Understanding Benchmark (ADU-Bench) to evaluate the performance of LALMs in the open-ended audio dialogue understanding.<n>ADU-Bench includes over 20,000 open-ended audio dialogues for the assessment of LALMs.
  arXiv  Detail & Related papers  (2024-12-06T16:34:15Z)
• Prosody in Cascade and Direct Speech-to-Text Translation: a case study on Korean Wh-Phrases [79.07111754406841]
  This work proposes using contrastive evaluation to measure the ability of direct S2TT systems to disambiguate utterances where prosody plays a crucial role. Our results clearly demonstrate the value of direct translation systems over cascade translation models.
  arXiv  Detail & Related papers  (2024-02-01T14:46:35Z)
• Exploring the Factual Consistency in Dialogue Comprehension of Large Language Models [51.75805497456226]
  This work focuses on the factual consistency issue with the help of the dialogue summarization task. Our evaluation shows that, on average, 26.8% of the summaries generated by LLMs contain factual inconsistency. To stimulate and enhance the dialogue comprehension ability of LLMs, we propose a fine-tuning paradigm with auto-constructed multi-task data.
  arXiv  Detail & Related papers  (2023-11-13T09:32:12Z)
• Continuous Speech Separation with Conformer [60.938212082732775]
  We use transformer and conformer in lieu of recurrent neural networks in the separation system. We believe capturing global information with the self-attention based method is crucial for the speech separation.
  arXiv  Detail & Related papers  (2020-08-13T09:36:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.