Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering

• URL: http://arxiv.org/abs/2606.11386v1
• Date: Tue, 09 Jun 2026 19:08:07 GMT
• Title: Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering
• Abstract summary: We analyze predictive behavior in full spoken language models (FDSLMs)<n>We show that FDSLMs dynamically modulate their internal focus between two states: a generative state aligned with model output generation and a perceptive state aligned incoming user input.<n>During user interruptions, the model remains transiently biased toward the generative state before transitioning into the perceptive state, causing it miss the beginning of incoming input.
• Score: 87.43777061308658
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Full-duplex spoken language models (FD-SLMs) enable seamless speech interaction by allowing models to listen and speak simultaneously, yet the internal mechanism by which they coordinate listening and speaking remains underexplored. We analyze the predictive behavior encoded in FD-SLM hidden representations and find that they exhibit stream-specific predictive patterns: during listening, they preferentially predict the incoming user stream, whereas during speaking, they preferentially predict the model output stream. Building on this observation, we show that FD-SLMs dynamically modulate their internal predictive focus between two states: a generative state aligned with model output generation and a perceptive state aligned with incoming user input. However, this modulation can lag behind abrupt changes in conversational context. During user interruptions, the model remains transiently biased toward the generative state before transitioning into the perceptive state, causing it to miss the beginning of the incoming input. We term this delayed internal transition state inertia. To quantify its downstream impact, we introduce the Zero-Buffer Benchmark (ZBB), a diagnostic benchmark for evaluating immediate interruption comprehension when user speech begins abruptly. We evaluate this setting using response correctness and initial-word occurrence rate (IWOR). Finally, we mitigate state inertia through activation steering with a perception vector, a training-free intervention with little additional computational overhead. Across multiple state-of-the-art FD-SLMs, activation steering substantially improves interruption handling; for example, on PersonaPlex, it improves correctness from 28% to 45% and IWOR from 40% to 72% without any fine-tuning.

Related papers

• Decoupling Turn-Taking from Semantics: A Decoupled Data Approach for Finite-State-Machine-Based Full-Duplex Dialogue [23.813275217960093]
  Large Models (LLMs) cannot faithfully simulate the fine-grained acoustic temporal dynamics of real human dialogues.<n>We propose a decoupled data approach that learns turn-taking from real Human-Human (HH) spoken dialogues while shaping behavior through semantic HumanAgent (HA) text dialogues.
  arXiv  Detail & Related papers  (2026-09-03T03:17:07Z)
• Prediction of Prediction (PoP): Inter-Layer Activation Fusion for Single-Pass Hallucination Detection in Large Language Models [0.0]
  This work evaluates whether internal hidden-state transition dynamics during generation can signal factual errors without auxiliary decoding calls.<n>We introduce Prediction of Prediction (PoP), a mechanism that captures layer-transition uncertainty.<n>PoP achieves an area under the receiver operating characteristic curve (AUROC) of 75.5% for factual-correctness classification.
  arXiv  Detail & Related papers  (2026-08-27T14:17:14Z)
• X2-Turn: Frame-Synchronous Dual-Head Modeling for Joint Streaming ASR and Turn State Prediction [6.446583777614073]
  X2-Turn is a frame-synchronous turn state prediction method via delayed-stream modeling.<n>We introduce a frame-synchronous turn state head that operates in parallel with the ASR head on shared streaming representations.
  arXiv  Detail & Related papers  (2026-08-11T12:54:52Z)
• ProactiveLLM: Learning Active Interaction for Streaming Large Language Models [19.8873622850537]
  We propose ProactiveLLM, which achieves active interaction by leveraging the model's endogenous states to guide interaction decisions.<n>ProactiveLLM significantly reduces interaction latency while maintaining quality, validating its capacity for dynamic and active interaction.
  arXiv  Detail & Related papers  (2026-05-30T04:31:29Z)
• Synchronization and Turn-Taking in Full-Duplex Speech Dialogue Models [3.5946669116828134]
  Full spoken dialogue models can speak simultaneously closer to human conversation than turn-based systems.<n>We study such models coordinate their internal interaction using neural coupling in human communication.<n>We find strong representational synchronization under no noise conditions, peaking near zero lag and degrading noise.
  arXiv  Detail & Related papers  (2026-05-19T18:11:03Z)
• Bi-Predictability: A Real-Time Signal for Monitoring LLM Interaction Integrity [0.0]
  We show that multi-turn interaction integrity can be continuously monitored using bi-predictability (P)<n>We introduce the Information Digital Twin (IDT), a lightweight architecture that estimates P across the context, response, next prompt loop without secondary inference or embeddings.
  arXiv  Detail & Related papers  (2026-03-18T18:10:37Z)
• Coherence in the brain unfolds across separable temporal regimes [1.3874648807526748]
  Coherence in language requires the brain to satisfy two competing temporal demands.<n>We show that coherence is implemented through dissociable neural regimes of slow contextual integration and rapid event-driven reconfiguration.
  arXiv  Detail & Related papers  (2025-12-23T16:16:42Z)
• Chronological Thinking in Full-Duplex Spoken Dialogue Language Models [66.84843878538207]
  Chronological Thinking aims to improve response quality in full SDLMs.<n>No additional latency: once the user stops speaking, the agent halts thinking and begins speaking without further delay.<n>Results: Experiments demonstrate the effectiveness of chronological thinking through both objective metrics and human evaluations.
  arXiv  Detail & Related papers  (2025-10-02T10:28:11Z)
• FLEXI: Benchmarking Full-duplex Human-LLM Speech Interaction [49.83226596963294]
  Speech-computer human interaction enables real-time spoken dialogue systems.<n>Modelling and benchmarking these models remains a fundamental challenge.<n>We introduce FLEXI, the first benchmark for full-human spoken interaction.
  arXiv  Detail & Related papers  (2025-09-26T11:57:42Z)
• Entropy-based Coarse and Compressed Semantic Speech Representation Learning [72.18542411704347]
  We propose an entropy-based dynamic aggregation framework for learning compressed semantic speech representations.<n> Experiments on ASR, speech-to-text translation, and voice conversion tasks demonstrate that the compressed representations perform on par with or better than dense token sequences.
  arXiv  Detail & Related papers  (2025-08-30T13:50:58Z)
• ARIG: Autoregressive Interactive Head Generation for Real-time Conversations [15.886402427095515]
  Face-to-face communication, as a common human activity, motivates the research on interactive head generation.<n>Previous clip-wise generation paradigm or explicit listener/speaker generator-switching methods have limitations in future signal acquisition.<n>We propose an autoregressive (AR) based frame-wise framework called ARIG to realize the real-time generation with better interaction realism.
  arXiv  Detail & Related papers  (2025-07-01T06:38:14Z)
• Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities [93.09944267871163]
  FullDuplexBench is a benchmark that systematically evaluates key interactive behaviors.<n>By releasing our benchmark code we aim to advance spoken dialogue modeling and the development of more natural and engaging SDMs.
  arXiv  Detail & Related papers  (2025-03-06T18:59:16Z)
• TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation [61.564874831498145]
  TranSpeech is a speech-to-speech translation model with bilateral perturbation. We establish a non-autoregressive S2ST technique, which repeatedly masks and predicts unit choices. TranSpeech shows a significant improvement in inference latency, enabling speedup up to 21.4x than autoregressive technique.
  arXiv  Detail & Related papers  (2022-05-25T06:34:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.