Protecting Bystander Privacy via Selective Hearing in Audio LLMs

• URL: http://arxiv.org/abs/2512.06380v2
• Date: Sat, 13 Dec 2025 14:48:07 GMT
• Title: Protecting Bystander Privacy via Selective Hearing in Audio LLMs
• Abstract summary: We introduce SH-Bench, the first benchmark designed to evaluate selective hearing.<n> SH-Bench contains 3,968 multi-speaker audio mixtures, including both real-world and synthetic scenarios.<n>We also present Bystander Privacy Fine-Tuning (BPFT), a novel training pipeline that teaches models to refuse bystander-related queries.
• Score: 14.82452941000742
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Audio Large language models (LLMs) are increasingly deployed in the real world, where they inevitably capture speech from unintended nearby bystanders, raising privacy risks that existing benchmarks and defences did not consider. We introduce SH-Bench, the first benchmark designed to evaluate selective hearing: a model's ability to attend to an intended main speaker while refusing to process or reveal information about incidental bystander speech. SH-Bench contains 3,968 multi-speaker audio mixtures, including both real-world and synthetic scenarios, paired with 77k multiple-choice questions that probe models under general and selective operating modes. In addition, we propose Selective Efficacy (SE), a novel metric capturing both multi-speaker comprehension and bystander-privacy protection. Our evaluation of state-of-the-art open-source and proprietary LLMs reveals substantial bystander privacy leakage, with strong audio understanding failing to translate into selective protection of bystander privacy. To mitigate this gap, we also present Bystander Privacy Fine-Tuning (BPFT), a novel training pipeline that teaches models to refuse bystander-related queries without degrading main-speaker comprehension. We show that BPFT yields substantial gains, achieving an absolute 47% higher bystander accuracy under selective mode and an absolute 16% higher SE compared to Gemini 2.5 Pro, which is the best audio LLM without BPFT. Together, SH-Bench and BPFT provide the first systematic framework for measuring and improving bystander privacy in audio LLMs.

Related papers

• Seeing Without Exposing: Adaptive Privacy Control for Open-World, Context-Hungry MLLMs [61.80513991207956]
  We propose Anchored Privacy Drifting (APD) to tackle privacy challenges in MLLMs.<n>APD drifts privacy-sensitive elements toward semantically equivalent alternatives while anchoring contextual cues to the source image.<n>We show that our method achieves balanced improvements in both privacy sanitization and content retention.
  arXiv  Detail & Related papers  (2026-06-05T11:41:39Z)
• VoxPrivacy: A Benchmark for Evaluating Interactional Privacy of Speech Language Models [25.266028200777317]
  Speech Language Models (SLMs) are expected to distinguish between users to manage information flow appropriately.<n>Current SLM benchmarks test dialogue ability but overlook speaker identity.<n>We introduce VoxPrivacy, the first benchmark designed to evaluate interactional privacy in SLMs.
  arXiv  Detail & Related papers  (2026-01-27T06:22:14Z)
• Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback [82.70507055599093]
  We present the first systematic study of preference learning for improving SDS quality in both multi-turn Chain-of-Thought and blockwise duplex models.<n>Experiments show that single-reward RLAIF selectively improves its targeted metric, while joint multi-reward training yields consistent gains across semantic quality and audio naturalness.
  arXiv  Detail & Related papers  (2026-01-27T00:55:14Z)
• HearSay Benchmark: Do Audio LLMs Leak What They Hear? [71.05839007164776]
  This paper investigates whether Audio Large Language Models inadvertently leak user privacy solely through acoustic voiceprints.<n>$textitHearSay$ is a comprehensive benchmark constructed from over 22,000 real-world audio clips.<n>Experiments on $textbfSignificant Privacy Leakage$ yield three critical findings.
  arXiv  Detail & Related papers  (2026-01-07T10:33:44Z)
• SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks [17.77094760401298]
  We study the vulnerability of fine-tuned large language models to membership inference attacks (MIAs)<n>We propose SOFT, a novel defense technique that mitigates privacy leakage by leveraging influential data selection with an adjustable parameter to balance utility preservation and privacy protection.
  arXiv  Detail & Related papers  (2025-06-12T07:23:56Z)
• Multi-P$^2$A: A Multi-perspective Benchmark on Privacy Assessment for Large Vision-Language Models [65.2761254581209]
  We evaluate the privacy preservation capabilities of 21 open-source and 2 closed-source Large Vision-Language Models (LVLMs)<n>Based on Multi-P$2$A, we evaluate the privacy preservation capabilities of 21 open-source and 2 closed-source LVLMs.<n>Our results reveal that current LVLMs generally pose a high risk of facilitating privacy breaches.
  arXiv  Detail & Related papers  (2024-12-27T07:33:39Z)
• On Active Privacy Auditing in Supervised Fine-tuning for White-Box Language Models [7.275432177367344]
  Parsing is designed to identify and quantify privacy leakage risks during the supervised fine-tuning of language models (LMs) We have improved the effectiveness of white-box membership inference attacks (MIAs) on large LMs including GPT-2, Llama2, and certain variants of them. Our research aims to provide the SFT community of LMs with a reliable, ready-to-use privacy auditing tool, and to offer valuable insights into safeguarding privacy during the fine-tuning process.
  arXiv  Detail & Related papers  (2024-11-11T15:46:07Z)
• Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning [62.224804688233]
  differential privacy (DP) offers a promising solution by ensuring models are 'almost indistinguishable' with or without any particular privacy unit. We study user-level DP motivated by applications where it necessary to ensure uniform privacy protection across users.
  arXiv  Detail & Related papers  (2024-06-20T13:54:32Z)
• Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory [82.7042006247124]
  We show that even the most capable AI models reveal private information in contexts that humans would not, 39% and 57% of the time, respectively. Our work underscores the immediate need to explore novel inference-time privacy-preserving approaches, based on reasoning and theory of mind.
  arXiv  Detail & Related papers  (2023-10-27T04:15:30Z)
• Crowdotic: A Privacy-Preserving Hospital Waiting Room Crowd Density Estimation with Non-speech Audio [4.149485024539117]
  We propose a non-speech audio-based approach for crowd analytics. Non-speech audio alone can be used to conduct such analysis with remarkable accuracy.
  arXiv  Detail & Related papers  (2023-09-19T03:08:20Z)
• Membership Inference Attacks Against Self-supervised Speech Models [62.73937175625953]
  Self-supervised learning (SSL) on continuous speech has started gaining attention. We present the first privacy analysis on several SSL speech models using Membership Inference Attacks (MIA) under black-box access.
  arXiv  Detail & Related papers  (2021-11-09T13:00:24Z)
• Research Challenges in Designing Differentially Private Text Generation Mechanisms [5.123298347655088]
  We describe some challenges in balancing the tradeoff between privacy and utility for differentially private text mechanisms. Our objective is not to evaluate a single solution but to further the conversation on these challenges and chart pathways for building better mechanisms.
  arXiv  Detail & Related papers  (2020-12-10T01:44:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.