HEAR Who Said What: Unlocking Speaker-Attributed Reasoning via Counterfactual Voice Grounding
Abstract Overview
This paper studies whether speech language models can correctly determine who said what in multi-speaker audio and reason over speaker identities. It introduces HEAR, a hierarchical benchmark for speaker-attributed reasoning with 2.4K human-verified multiple-choice samples from 887 real-world audio clips, spanning discrimination, attribution, and reasoning tasks. Evaluations of 20 leading speech language models show that many models perform poorly on foundational speaker attribution and often rely on semantic priors instead of acoustic vocal cues. To address this, the authors build CASH, a 60K-scale counterfactual audio dataset with speaker-level hard negatives, and train A2R, a 30B model that first produces a speaker-tagged transcript and then reasons over it.
Novelty
The main novelty is the explicit framing and diagnosis of speaker-attributed reasoning as a hierarchical capability, operationalized through the new HEAR benchmark. The work is also distinctive in using counterfactual voice grounding via CASH, where the transcript is preserved but the speaker identity is swapped, forcing models to depend on vocal evidence rather than language priors.
Results
On HEAR, A2R outperforms the open-source baselines by a large margin, reaching a 67.1 average score versus 35.4 for its Qwen3-Omni-30B-A3B-Instruct base model, and it attains much stronger paired reasoning accuracy that tests resistance to semantic hallucination. The model also shows zero-shot transfer gains on speaker-attribution-critical tasks, improving over the base model by +36.4 on WDYL, +22.6 on GAOKAO, and +19.0 on FTS. These gains persist on corresponding human-recorded versions of the tasks, while causing only modest degradation on general dyadic VoiceBench evaluations.
Key Points
- HEAR provides a structured benchmark for evaluating multi-speaker understanding across discrimination, attribution, and reasoning, with human-verified examples from real-world audio.
- The study shows that many current speech language models overestimate their reasoning ability because they often answer from semantic plausibility rather than true voice-grounded speaker attribution.
- A2R, trained on the CASH counterfactual dataset with a transcription-first objective, improves speaker-aware reasoning and transfers to unseen attribution-critical tasks, including human-recorded audio.