VoiceBBQ: Investigating Effect of Content and Acoustics in Social Bias of Spoken Language Model

• URL: http://arxiv.org/abs/2509.21108v1
• Date: Thu, 25 Sep 2025 12:54:31 GMT
• Title: VoiceBBQ: Investigating Effect of Content and Acoustics in Social Bias of Spoken Language Model
• Abstract summary: VoiceBBQ is a dataset that measures social bias by presenting ambiguous or disambiguated contexts followed by questions that may elicit stereotypical responses.<n>The dataset converts every BBQ context into controlled voice conditions, enabling per-axis accuracy, bias, and consistency scores that remain comparable to the original text benchmark.
• Score: 3.8581904575338886
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce VoiceBBQ, a spoken extension of the BBQ (Bias Benchmark for Question Answering) - a dataset that measures social bias by presenting ambiguous or disambiguated contexts followed by questions that may elicit stereotypical responses. Due to the nature of speech, social bias in Spoken Language Models (SLMs) can emerge from two distinct sources: 1) content aspect and 2) acoustic aspect. The dataset converts every BBQ context into controlled voice conditions, enabling per-axis accuracy, bias, and consistency scores that remain comparable to the original text benchmark. Using VoiceBBQ, we evaluate two SLMs - LLaMA-Omni and Qwen2-Audio - and observe architectural contrasts: LLaMA-Omni resists acoustic bias while amplifying gender and accent bias, whereas Qwen2-Audio substantially dampens these cues while preserving content fidelity. VoiceBBQ thus provides a compact, drop-in testbed for jointly diagnosing content and acoustic bias across spoken language models.

Related papers

• SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models [42.27807245438137]
  We introduce textscSpeechEQ, a comprehensive framework designed to evaluate the sociolinguistic reasoning of Speech-Language Models (SLMs)<n>The framework includes a validated dataset of 2,265 dialogues across 15 Emotional Quotient (EQ) subscales grounded in EQ-i 2.0 theory.
  arXiv  Detail & Related papers  (2026-06-24T16:03:38Z)
• Afrispeech Semantics: Evaluating Audio Semantic Reasoning in Spoken Language Models Across Domains and Accents [3.6734440150955368]
  Audio language models (ALMs) are increasingly used for speech-based understanding.<n>The effects of accent variation, domain shift, and semantic over-inference on audio reasoning are poorly understood.<n>We evaluate audio language models across five semantic and paralinguistic reasoning tasks.
  arXiv  Detail & Related papers  (2026-05-11T20:27:40Z)
• VoxSafeBench: Not Just What Is Said, but Who, How, and Where [30.964672315074413]
  We introduce VoxSafeBench, among the first benchmarks to jointly evaluate social alignment in speech language models.<n>Tier 1 evaluates content-centric risks using matched text and audio inputs.<n>Tier 2 targets audio-conditioned risks in which the transcript is benign but the appropriate response hinges on the speaker.<n>We find that safeguards appearing robust on text often degrade in speech.
  arXiv  Detail & Related papers  (2026-04-16T02:24:59Z)
• AEQ-Bench: Measuring Empathy of Omni-Modal Large Models [55.722881748046895]
  We introduce AEQ-Bench, a novel benchmark to assess two core empathetic capabilities of omni-modal large models (OLMs)<n>AEQ-Bench incorporates two novel settings that vary in context specificity and speech tone.<n> Comprehensive assessment across linguistic and paralinguistic metrics reveals that OLMs trained with audio output capabilities generally outperformed models with text-only outputs.
  arXiv  Detail & Related papers  (2026-01-15T15:39:50Z)
• Speak Your Mind: The Speech Continuation Task as a Probe of Voice-Based Model Bias [24.932603485660323]
  Speech Continuation (SC) is the task of generating a coherent extension of a spoken prompt while preserving semantic context and speaker identity.<n>We present the first systematic evaluation of bias in SC, investigating how gender and phonation type affect continuation behaviour.
  arXiv  Detail & Related papers  (2025-09-26T08:43:25Z)
• AHELM: A Holistic Evaluation of Audio-Language Models [78.20477815156484]
  multimodal audio-language models (ALMs) take interleaved audio and text as input and output text.<n>AHELM is a benchmark that aggregates various datasets -- including 2 new synthetic audio-text datasets called PARADE and CoRe-Bench.<n>We also standardize the prompts, inference parameters, and evaluation metrics to ensure equitable comparisons across models.
  arXiv  Detail & Related papers  (2025-08-29T07:40:39Z)
• SpeechIQ: Speech Intelligence Quotient Across Cognitive Levels in Voice Understanding Large Language Models [76.07833875692722]
  Speech-based Intelligence Quotient (SIQ) is a new form of human cognition-inspired evaluation pipeline for voice understanding large language models.<n>Our framework represents a first-of-its-kind intelligence examination that bridges cognitive principles with voice-oriented benchmarks.
  arXiv  Detail & Related papers  (2025-07-25T15:12:06Z)
• Multi-Domain Audio Question Answering Toward Acoustic Content Reasoning in The DCASE 2025 Challenge [102.84031769492708]
  This task defines three QA subsets to test audio-language models on interactive question-answering over diverse acoustic scenes.<n>Preliminary results on the development set are compared, showing strong variation across models and subsets.<n>This challenge aims to advance the audio understanding and reasoning capabilities of audio-language models toward human-level acuity.
  arXiv  Detail & Related papers  (2025-05-12T09:04:16Z)
• Solla: Towards a Speech-Oriented LLM That Hears Acoustic Context [45.56363286769136]
  We introduce Solla, a novel framework designed to understand speech-based questions and hear the acoustic context concurrently.<n>Solla incorporates an audio tagging module to effectively identify and represent audio events, as well as an ASR-assisted prediction method to improve comprehension of spoken content.<n>We propose a new benchmark dataset called SA-Eval, which includes three tasks: audio event classification, audio captioning, and audio question answering.
  arXiv  Detail & Related papers  (2025-03-19T15:34:21Z)
• VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models [32.086847480051084]
  We present VoxEval, a novel SpeechQA benchmark that assesses knowledge understanding through pure speech interactions.<n>Our benchmark 1) maintains speech format for both inputs and outputs, 2) evaluates model robustness across diverse input audio conditions, and 3) pioneers the assessment of complex tasks like mathematical reasoning in spoken format.
  arXiv  Detail & Related papers  (2025-01-09T04:30:12Z)
• Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialogue [41.10328851671422]
  SpeechLLMs have demonstrated impressive spoken dialog question-answering (SQA) performance in benchmarks like Gaokao. We show that SpeechLLMs exhibit limited speaker awareness from the audio and behave similarly to an LLM reasoning from the conversation transcription without sound. We propose that tasks focused on identity-critical questions could offer a more accurate evaluation framework of SpeechLLMs in SQA.
  arXiv  Detail & Related papers  (2024-09-07T22:54:47Z)
• Listen and Speak Fairly: A Study on Semantic Gender Bias in Speech Integrated Large Language Models [38.64792118903994]
  We evaluate gender bias in SILLMs across four semantic-related tasks. Our analysis reveals that bias levels are language-dependent and vary with different evaluation methods.
  arXiv  Detail & Related papers  (2024-07-09T15:35:43Z)
• AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension [95.8442896569132]
  We introduce AIR-Bench, the first benchmark to evaluate the ability of Large Audio-Language Models (LALMs) to understand various types of audio signals and interact with humans in the textual format. Results demonstrate a high level of consistency between GPT-4-based evaluation and human evaluation.
  arXiv  Detail & Related papers  (2024-02-12T15:41:22Z)
• DUAL: Textless Spoken Question Answering with Speech Discrete Unit Adaptive Learning [66.71308154398176]
  Spoken Question Answering (SQA) has gained research attention and made remarkable progress in recent years. Existing SQA methods rely on Automatic Speech Recognition (ASR) transcripts, which are time and cost-prohibitive to collect. This work proposes an ASR transcript-free SQA framework named Discrete Unit Adaptive Learning (DUAL), which leverages unlabeled data for pre-training and is fine-tuned by the SQA downstream task.
  arXiv  Detail & Related papers  (2022-03-09T17:46:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.