The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning

• URL: http://arxiv.org/abs/2603.17837v1
• Date: Wed, 18 Mar 2026 15:30:29 GMT
• Title: The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning
• Abstract summary: Internal cognitive processing is instrumental in formulating high-quality responses.<n>We propose a novel method named FLAIR that conducts latent thinking simultaneously with speech perception.<n>Our approach aligns seamlessly with spoken dialogue systems.
• Score: 76.96796481909581
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: During conversational interactions, humans subconsciously engage in concurrent thinking while listening to a speaker. Although this internal cognitive processing may not always manifest as explicit linguistic structures, it is instrumental in formulating high-quality responses. Inspired by this cognitive phenomenon, we propose a novel Full-duplex LAtent and Internal Reasoning method named FLAIR that conducts latent thinking simultaneously with speech perception. Unlike conventional "thinking" mechanisms in NLP, which require post-hoc generation, our approach aligns seamlessly with spoken dialogue systems: during the user's speaking phase, it recursively feeds the latent embedding output from the previous step into the next step, enabling continuous reasoning that strictly adheres to causality without introducing additional latency. To enable this latent reasoning, we design an Evidence Lower Bound-based objective that supports efficient supervised finetuning via teacher forcing, circumventing the need for explicit reasoning annotations. Experiments demonstrate the effectiveness of this think-while-listening design, which achieves competitive results on a range of speech benchmarks. Furthermore, FLAIR robustly handles conversational dynamics and attains competitive performance on full-duplex interaction metrics.

Related papers

• Don't Wait to Reply: Towards Responsive yet Thoughtful Dialogue through Proactive Thinking [42.788336763990316]
  Thinking has emerged as a critical capability for Large Language Models tackling complex tasks.<n>We propose Proactive Thinking, a framework that empowers models to pre-compute potential response elements during conversational downtime.<n>We demonstrate that proactive thinking effectively improves interaction efficiency without compromising performance.
  arXiv  Detail & Related papers  (2026-07-03T08:24:23Z)
• LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning [31.98142661908727]
  We argue that a unified latent space is a better medium for such reasoning because it preserves dense sensory information while remaining compatible with autoregressive generation.<n>Based on this insight, we propose textbfLatent Omni, a cross-modal reasoning framework that interleaves textual reasoning with audio-visual latent states.
  arXiv  Detail & Related papers  (2026-05-21T05:18:57Z)
• Conversational Behavior Modeling Foundation Model With Multi-Level Perception [13.659870465634228]
  We introduce framework models that predict intent and then reasons over conversational behaviors via a Graph-of-Thoughts (GoT)<n>GoT structures streaming predictions as an evolving graph, enabling a transformer to forecast the next speech act generate concise justifications for its decisions.<n>Experiments show that the framework delivers robust behavior detection, produces interpretable reasoning chains, and establishes a foundation for benchmarking conversational reasoning in full duplex spoken dialogue systems.
  arXiv  Detail & Related papers  (2026-02-11T17:32:52Z)
• A Unified Spoken Language Model with Injected Emotional-Attribution Thinking for Human-like Interaction [50.05919688888947]
  This paper presents a unified spoken language model for emotional intelligence, enhanced by a novel data construction strategy termed Injected Emotional-Attribution Thinking (IEAT)<n>IEAT incorporates user emotional states and their underlying causes into the model's internal reasoning process, enabling emotion-aware reasoning to be internalized rather than treated as explicit supervision.<n> Experiments on the Human-like Spoken Dialogue Systems Challenge (HumDial) Emotional Intelligence benchmark demonstrate that the proposed approach achieves top-ranked performance across emotional trajectory modeling, emotional reasoning, and empathetic response generation.
  arXiv  Detail & Related papers  (2026-01-08T14:07:30Z)
• Enabling Conversational Behavior Reasoning Capabilities in Full-Duplex Speech [15.41279444168073]
  We introduce framework enabling reasoning over conversational behaviors by modeling this process as causal inference within a Graph ofThoughts (GoT)<n>We develop a hybrid corpus that pairs controllable, event-rich simulations with humanannotated rationales and real conversational speech.<n>GoT framework structures streaming predictions as an evolving graph, enabling a multimodal transformer to forecast the next speech act.
  arXiv  Detail & Related papers  (2025-12-25T15:00:50Z)
• Chronological Thinking in Full-Duplex Spoken Dialogue Language Models [66.84843878538207]
  Chronological Thinking aims to improve response quality in full SDLMs.<n>No additional latency: once the user stops speaking, the agent halts thinking and begins speaking without further delay.<n>Results: Experiments demonstrate the effectiveness of chronological thinking through both objective metrics and human evaluations.
  arXiv  Detail & Related papers  (2025-10-02T10:28:11Z)
• Mini-Omni-Reasoner: Token-Level Thinking-in-Speaking in Large Speech Models [80.75260664100644]
  Mini-Omni-Reasoner is a framework that enables reasoning within speech via a novel "Thinking-in-Speaking" formulation.<n>It interleaves silent reasoning tokens with spoken response tokens at the token level.<n>It achieves a +19.1% gain in arithmetic reasoning and +6.4% in contextual understanding, with shorter outputs and zero decoding latency.
  arXiv  Detail & Related papers  (2025-08-18T15:14:04Z)
• Multi-dimensional Evaluation of Empathetic Dialog Responses [4.580983642743026]
  We propose a multi-dimensional empathy evaluation framework to measure both emphexpressed intents from the speaker's perspective and emphperceived empathy from the listener's perspective. We find the two dimensions are inter-connected, while perceived empathy has high correlations with dialogue satisfaction levels.
  arXiv  Detail & Related papers  (2024-02-18T00:32:33Z)
• MindDial: Belief Dynamics Tracking with Theory-of-Mind Modeling for Situated Neural Dialogue Generation [62.44907105496227]
  MindDial is a novel conversational framework that can generate situated free-form responses with theory-of-mind modeling. We introduce an explicit mind module that can track the speaker's belief and the speaker's prediction of the listener's belief. Our framework is applied to both prompting and fine-tuning-based models, and is evaluated across scenarios involving both common ground alignment and negotiation.
  arXiv  Detail & Related papers  (2023-06-27T07:24:32Z)
• You Impress Me: Dialogue Generation via Mutual Persona Perception [62.89449096369027]
  The research in cognitive science suggests that understanding is an essential signal for a high-quality chit-chat conversation. Motivated by this, we propose P2 Bot, a transmitter-receiver based framework with the aim of explicitly modeling understanding.
  arXiv  Detail & Related papers  (2020-04-11T12:51:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.