VoxSafeBench: Not Just What Is Said, but Who, How, and Where

• URL: http://arxiv.org/abs/2604.14548v2
• Date: Mon, 20 Apr 2026 07:51:45 GMT
• Title: VoxSafeBench: Not Just What Is Said, but Who, How, and Where
• Abstract summary: We introduce VoxSafeBench, among the first benchmarks to jointly evaluate social alignment in speech language models.<n>Tier 1 evaluates content-centric risks using matched text and audio inputs.<n>Tier 2 targets audio-conditioned risks in which the transcript is benign but the appropriate response hinges on the speaker.<n>We find that safeguards appearing robust on text often degrade in speech.
• Score: 30.964672315074413
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As speech language models (SLMs) transition from personal devices into shared, multi-user environments, their responses must account for far more than the words alone. Who is speaking, how they sound, and where the conversation takes place can each turn an otherwise benign request into one that is unsafe, unfair, or privacy-violating. Existing benchmarks, however, largely focus on basic audio comprehension, study individual risks in isolation, or conflate content that is inherently harmful with content that only becomes problematic due to its acoustic context. We introduce VoxSafeBench, among the first benchmarks to jointly evaluate social alignment in SLMs across three dimensions: safety, fairness, and privacy. VoxSafeBench adopts a Two-Tier design: Tier1 evaluates content-centric risks using matched text and audio inputs, while Tier2 targets audio-conditioned risks in which the transcript is benign but the appropriate response hinges on the speaker, paralinguistic cues, or the surrounding environment. To validate Tier2, we include intermediate perception probes and confirm that frontier SLMs can successfully detect these acoustic cues yet still fail to act on them appropriately. Across 22 tasks with bilingual coverage, we find that safeguards appearing robust on text often degrade in speech: safety awareness drops for speaker- and scene-conditioned risks, fairness erodes when demographic differences are conveyed vocally, and privacy protections falter when contextual cues arrive acoustically. Together, these results expose a pervasive speech grounding gap: current SLMs frequently recognize the relevant social norm in text but fail to apply it when the decisive cue must be grounded in speech. Code and data are publicly available at: https://amphionteam.github.io/VoxSafeBench_demopage/

Related papers

• AudioGuard: Toward Comprehensive Audio Safety Protection Across Diverse Threat Models [17.541986184072773]
  Real-world risks can hinge on audio-native harmful sound events, speaker attributes, impersonation/voice-cloning misuse.<n>AudioGuard is a unified guardrail consisting of 1) SoundGuard for waveform-level audio-native detection and 2) ContentGuard for policy-grounded semantic protection.
  arXiv  Detail & Related papers  (2026-04-10T02:02:12Z)
• Synthetic Voices, Real Threats: Evaluating Large Text-to-Speech Models in Generating Harmful Audio [63.18443674004945]
  This work explores a content-centric threat: exploiting TTS systems to produce speech containing harmful content.<n>We present HARMGEN, a suite of five attacks organized into two families that address these challenges.
  arXiv  Detail & Related papers  (2025-11-14T03:00:04Z)
• Speech-Audio Compositional Attacks on Multimodal LLMs and Their Mitigation with SALMONN-Guard [37.736305528135944]
  We introduce SACRED-Bench to evaluate robustness of large language models (LLMs) under complex audio-based attacks.<n>We propose SALMONN-Guard, a safeguard LLM that jointly inspects speech, audio, and text for safety judgments.
  arXiv  Detail & Related papers  (2025-11-13T11:50:54Z)
• VoiceBBQ: Investigating Effect of Content and Acoustics in Social Bias of Spoken Language Model [3.8581904575338886]
  VoiceBBQ is a dataset that measures social bias by presenting ambiguous or disambiguated contexts followed by questions that may elicit stereotypical responses.<n>The dataset converts every BBQ context into controlled voice conditions, enabling per-axis accuracy, bias, and consistency scores that remain comparable to the original text benchmark.
  arXiv  Detail & Related papers  (2025-09-25T12:54:31Z)
• AHELM: A Holistic Evaluation of Audio-Language Models [78.20477815156484]
  multimodal audio-language models (ALMs) take interleaved audio and text as input and output text.<n>AHELM is a benchmark that aggregates various datasets -- including 2 new synthetic audio-text datasets called PARADE and CoRe-Bench.<n>We also standardize the prompts, inference parameters, and evaluation metrics to ensure equitable comparisons across models.
  arXiv  Detail & Related papers  (2025-08-29T07:40:39Z)
• AudioTrust: Benchmarking the Multifaceted Trustworthiness of Audio Large Language Models [125.44235244336623]
  Audio Large Language Models (ALLMs) have gained widespread adoption, yet their trustworthiness remains underexplored.<n>We identify significant trustworthiness risks in ALLMs arising from non-semantic acoustic cues, including timbre, accent, and background noise.<n>We propose AudioTrust, a framework for systematic evaluation of ALLM trustworthiness across audio-specific risks.
  arXiv  Detail & Related papers  (2025-05-22T04:27:46Z)
• VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models [32.086847480051084]
  We present VoxEval, a novel SpeechQA benchmark that assesses knowledge understanding through pure speech interactions.<n>Our benchmark 1) maintains speech format for both inputs and outputs, 2) evaluates model robustness across diverse input audio conditions, and 3) pioneers the assessment of complex tasks like mathematical reasoning in spoken format.
  arXiv  Detail & Related papers  (2025-01-09T04:30:12Z)
• Audio Is the Achilles' Heel: Red Teaming Audio Large Multimodal Models [50.89022445197919]
  We show that open-source audio LMMs suffer an average attack success rate of 69.14% on harmful audio questions. Our speech-specific jailbreaks on Gemini-1.5-Pro achieve an attack success rate of 70.67% on the harmful query benchmark.
  arXiv  Detail & Related papers  (2024-10-31T12:11:17Z)
• Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition [52.71604809100364]
  We propose wav2vec-Switch, a method to encode noise robustness into contextualized representations of speech. Specifically, we feed original-noisy speech pairs simultaneously into the wav2vec 2.0 network. In addition to the existing contrastive learning task, we switch the quantized representations of the original and noisy speech as additional prediction targets.
  arXiv  Detail & Related papers  (2021-10-11T00:08:48Z)
• Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining [64.35907499990455]
  We propose a framework to learn semantics directly from speech with semi-supervision from transcribed or untranscribed speech. Our framework is built upon pretrained end-to-end (E2E) ASR and self-supervised language models, such as BERT. In parallel, we identify two essential criteria for evaluating SLU models: environmental noise-robustness and E2E semantics evaluation.
  arXiv  Detail & Related papers  (2020-10-26T18:21:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.