Causal Analysis and Mitigation of Spurious Onsets in Full-Duplex Speech LLMs

• URL: http://arxiv.org/abs/2609.13445v1
• Date: Fri, 11 Sep 2026 19:03:36 GMT
• Title: Causal Analysis and Mitigation of Spurious Onsets in Full-Duplex Speech LLMs
• Abstract summary: We find that speech probability spikes by over nine orders of magnitude in one 80-ms frame.<n>Our method suppresses 13/13 Moshi and 9/9 PersonaPlex spurious onsets, while preserving 40 genuine responses per model.<n>Our method requires no retraining and runs in real-time, with 95th-percentile decision time below 61 ms, within the 80-ms frame budget.
• Score: 3.502729154227436
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Speech-to-speech LLMs like Moshi, and its derivative PersonaPlex, can listen and speak concurrently through full-duplex generation. However, they can begin speaking inappropriately during prolonged user silence: under digital-zero input, Moshi and PersonaPlex initiate speech in 12/40 and 11/40 five-minute continuations, respectively. What causes this spurious speech? We investigate two hypotheses: either repeated sampling selects speech despite persistently low onset probabilities, or conditioning on the model's nonspeech outputs causes an abrupt spike in onset probability. We find that, at every observed onset, speech probability spikes by over nine orders of magnitude in one 80-ms frame, supporting the latter hypothesis. Then, to suppress these onsets without blocking genuine responses, we ask a causal counterfactual question: is the model responding to user speech, or would its next-token distribution remain similar if the preceding user input were muted? Accordingly, we suppress onsets whose distributions change little under this intervention. Across 40 held-out trials per model with realistic microphone noise, our method suppresses 13/13 Moshi and 9/9 PersonaPlex spurious onsets, while preserving 40/40 genuine responses per model. Our inference-time method requires no retraining and runs in real-time, with 95th-percentile decision time below 61 ms, within the 80-ms frame budget. Our code is available at https://github.com/KentoNishi/icassp27-spurious-onsets.

Related papers

• DuplexJail: Safety Alignment Breaks Under Spoken Interruption in Full-Duplex Models [16.95513039797219]
  Fixed-delay interruption raises whole-response attack success rates on AdvBench to 40.3% for PersonaPlex and 48.7% for PersonaPlexRL.<n>The refusal policy reaches 35.6% and 48.6%, respectively, with all trials scored regardless of whether an interruption occurs.<n>These findings identify spoken interruption as a jailbreak attack evaluating safety throughout ongoing full interaction.
  arXiv  Detail & Related papers  (2026-09-08T20:19:24Z)
• Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model [0.23332469289621785]
  We train an audio-native interface for DiffusionGemma that generates text by uniform, random-token discrete diffusion.<n>We find that the natural training objectives fail to ground the audio because their gradient reaches the projector only through attention that has already dismissed it.<n>The resulting model reaches 6.6 percent word error rate on LibriSpeech test-clean, transcribes in roughly eight parallel steps regardless of utterance length, and uses a single adapter trained on six languages.
  arXiv  Detail & Related papers  (2026-07-14T17:53:22Z)
• Language models recognize dropout and Gaussian noise applied to their activations [74.06294367754748]
  We verbalize either (a) emphmask activations, simulating emphdropout, or (b) add emphGaussian noise to them.<n>We test models from the Llama, Olmo, and Qwen families, with sizes between 8B and 32B.
  arXiv  Detail & Related papers  (2026-04-19T14:30:13Z)
• DiffuSpeech: Silent Thought, Spoken Answer via Unified Speech-Text Diffusion [23.01044837428522]
  We introduce textbfSilent Thought, Spoken Answer'' -- a paradigm where speech LLMs generate internal text reasoning alongside spoken responses.<n>We present method, the first diffusion-based speech-text language model supporting both understanding and generation.<n>Experiments show method achieves state-of-the-art speech-to-speech QA accuracy, outperforming the best baseline by up to 9 points.
  arXiv  Detail & Related papers  (2026-01-30T12:08:33Z)
• SHANKS: Simultaneous Hearing and Thinking for Spoken Language Models [158.18422855768756]
  Current large language models (LLMs) and spoken language models (SLMs) begin thinking and taking actions only after the user has finished their turn.<n>This prevents the model from interacting during the user's turn and can lead to high response latency while it waits to think.<n>We propose SHANKS, a framework that enables SLMs to generate unspoken chain-of-thought reasoning while listening to the user input.
  arXiv  Detail & Related papers  (2025-10-08T11:48:59Z)
• STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Models [131.90117151306993]
  Spoken Language Models (SLMs) are designed to take speech inputs and produce spoken responses.<n>Current SLMs lack the ability to perform an internal, unspoken thinking process before responding.<n>We propose Stitch, a novel generation method that alternates between the generation of unspoken reasoning chunks and spoken response chunks.
  arXiv  Detail & Related papers  (2025-07-21T08:30:03Z)
• SpeechPrune: Context-aware Token Pruning for Speech Information Retrieval [23.055339472773596]
  We introduce Speech Information Retrieval (SIR), a new long-context task for Speech Large Language Models (Speech LLMs)<n>We present SPIRAL, a benchmark testing models' ability to extract critical details from 90-second spoken inputs.<n> SpeechPrune is a training-free token pruning strategy that uses speech-text similarity and approximated attention scores to efficiently discard irrelevant tokens.
  arXiv  Detail & Related papers  (2024-12-16T17:36:02Z)
• Moshi: a speech-text foundation model for real-time dialogue [78.88479749811376]
  Current systems for spoken dialogue rely on pipelines independent voice activity detection and text-to-speech. We show how Moshi Moshi can provide streaming speech recognition and text-to-speech. Our resulting model is first real-time full spoken large language model modality.
  arXiv  Detail & Related papers  (2024-09-17T17:55:39Z)
• TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation [61.564874831498145]
  TranSpeech is a speech-to-speech translation model with bilateral perturbation. We establish a non-autoregressive S2ST technique, which repeatedly masks and predicts unit choices. TranSpeech shows a significant improvement in inference latency, enabling speedup up to 21.4x than autoregressive technique.
  arXiv  Detail & Related papers  (2022-05-25T06:34:14Z)
• Improving Unsupervised Sparsespeech Acoustic Models with Categorical Reparameterization [31.977418525076626]
  We extend the Sparsespeech model to allow for sampling over a random variable, yielding pseudo-posteriorgrams. The new and improved model is trained and evaluated on the Libri-Light corpus, a benchmark for ASR with limited or no supervision. We observe a relative improvement of up to 31.4% on ABX error rates across speakers on the test set with the improved model.
  arXiv  Detail & Related papers  (2020-05-29T13:58:36Z)
• Listen Attentively, and Spell Once: Whole Sentence Generation via a Non-Autoregressive Architecture for Low-Latency Speech Recognition [66.47000813920619]
  We propose a non-autoregressive end-to-end speech recognition system called LASO. Because of the non-autoregressive property, LASO predicts a textual token in the sequence without the dependence on other tokens. We conduct experiments on publicly available Chinese dataset AISHELL-1.
  arXiv  Detail & Related papers  (2020-05-11T04:45:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.