Privacy-Preserving End-to-End Full-Duplex Speech Dialogue Models

• URL: http://arxiv.org/abs/2603.08179v1
• Date: Mon, 09 Mar 2026 10:01:24 GMT
• Title: Privacy-Preserving End-to-End Full-Duplex Speech Dialogue Models
• Abstract summary: We show that SALM-Duplex leakage persists across all layers with SALM-Duplex showing stronger leakage in early layers while Moshi leaks uniformly, and that Linkability rises sharply within the first few turns.<n>We propose two streaming anonymization setups using StreamVoiceAnon: a front-end (AnonW2W) and a feature-domain replacement (AnonW2F)
• Score: 62.35104196350636
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: End-to-end full-duplex speech models feed user audio through an always-on LLM backbone, yet the speaker privacy implications of their hidden representations remain unexamined. Following the VoicePrivacy 2024 protocol with a lazy-informed attacker, we show that the hidden states of SALM-Duplex and Moshi leak substantial speaker identity across all transformer layers. Layer-wise and turn-wise analyses reveal that leakage persists across all layers, with SALM-Duplex showing stronger leakage in early layers while Moshi leaks uniformly, and that Linkability rises sharply within the first few turns. We propose two streaming anonymization setups using Stream-Voice-Anon: a waveform-level front-end (Anon-W2W) and a feature-domain replacement (Anon-W2F). Anon-W2F raises EER by over 3.5x relative to the discrete encoder baseline (11.2% to 41.0%), approaching the 50% random-chance ceiling, while Anon-W2W retains 78-93% of baseline sBERT across setups with sub-second response latency (FRL under 0.8 s).

Related papers

• DuplexJail: Safety Alignment Breaks Under Spoken Interruption in Full-Duplex Models [16.95513039797219]
  Fixed-delay interruption raises whole-response attack success rates on AdvBench to 40.3% for PersonaPlex and 48.7% for PersonaPlexRL.<n>The refusal policy reaches 35.6% and 48.6%, respectively, with all trials scored regardless of whether an interruption occurs.<n>These findings identify spoken interruption as a jailbreak attack evaluating safety throughout ongoing full interaction.
  arXiv  Detail & Related papers  (2026-09-08T20:19:24Z)
• Enabling Proactive Spoken Turns via a Generalized Style-Aware Full-Duplex Framework [51.969792455913655]
  2 half dialogue systems system system waits for user turn completion before responds.<n>This creates key challenge: anticipate turn timing without sacrificing response quality.<n>We present WildTurnTurn, a largescale, real-world English dataset containing approximately981 hours of multi-turn conversations from face-to-face and telephone conversations, with five turn-level interaction styles, trained on WildTurn LPS-TC rich spoken dynamics that exhibit rich spoken dynamics that are not captured by existing static benchmarks, including we introduce a two-tier evaluation scheme that assesses both in-level timing precision and turn-level interaction quality under realistic streaming constraints,
  arXiv  Detail & Related papers  (2026-08-04T08:22:50Z)
• Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs [58.719964942364356]
  Lychee-FD is a native end-to-end full-to-end full SLMs framework.<n>Lychee-FD decouples conflicting modalities while preserving cross-modality coherence.<n>Our method significantly remains the state of the art.
  arXiv  Detail & Related papers  (2026-07-07T17:43:36Z)
• BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM [37.246890931249496]
  BayLing-Duplex is a native full-generation SpeechLM that decides where to listen, when to speak, and sacrifice when to stop with no auxiliary turn-taking module.<n>It reaches 92% turn-taking success and 100% interruption success on InstructS2S-Eval, while improving speech-response score from 2.17 to 3.39 over Moshi.
  arXiv  Detail & Related papers  (2026-06-12T15:01:51Z)
• StreamVoiceAnon+: Emotion-Preserving Streaming Speaker Anonymization via Frame-Level Acoustic Distillation [56.49717639074325]
  We address the challenge of preserving emotional content in streaming speaker anonymization (SA)<n>We propose supervised finetuning with neutral-emotion utterance pairs from the same speaker, combined with frame-level emotion distillation on acoustic token hidden states.<n>On the VoicePrivacy 2024 protocol, our approach achieves a 49.2% UAR (emotion preservation) with 5.77% WER (intelligibility)
  arXiv  Detail & Related papers  (2026-03-06T09:30:20Z)
• Stream-Voice-Anon: Enhancing Utility of Real-Time Speaker Anonymization via Neural Audio Codec and Language Models [51.7170633585748]
  Stream-Voice-Anon adapts modern causal LM-based NAC architectures specifically for streaming speaker anonymization.<n>Our anonymization approach incorporates pseudo-speaker representation sampling, a speaker embedding mixing and diverse prompt selection strategies.<n>Under the VoicePrivacy 2024 Challenge protocol, Stream-Voice-Anon achieves substantial improvements in intelligibility.
  arXiv  Detail & Related papers  (2026-01-20T13:23:44Z)
• Protecting Bystander Privacy via Selective Hearing in Audio LLMs [14.82452941000742]
  We introduce SH-Bench, the first benchmark designed to evaluate selective hearing.<n> SH-Bench contains 3,968 multi-speaker audio mixtures, including both real-world and synthetic scenarios.<n>We also present Bystander Privacy Fine-Tuning (BPFT), a novel training pipeline that teaches models to refuse bystander-related queries.
  arXiv  Detail & Related papers  (2025-12-06T10:24:04Z)
• Backdoor Attacks Against Speech Language Models [63.07317091368079]
  We present the first systematic study of audio backdoor attacks against speech language models.<n>We demonstrate its effectiveness across four speech encoders and three datasets, covering four tasks.<n>We propose a fine-tuning-based defense that mitigates the threat of poisoned pretrained encoders.
  arXiv  Detail & Related papers  (2025-10-01T17:45:04Z)
• VoxGuard: Evaluating User and Attribute Privacy in Speech via Membership Inference Attacks [51.68795949691009]
  We introduce VoxGuard, a framework grounded in differential privacy and membership inference.<n>For attributes, we show that simple transparent attacks recover gender and accent with near-perfect accuracy even after anonymization.<n>Our results demonstrate that EER substantially underestimates leakage, highlighting the need for low-FPR evaluation.
  arXiv  Detail & Related papers  (2025-09-22T20:57:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.