X2-Turn: Frame-Synchronous Dual-Head Modeling for Joint Streaming ASR and Turn State Prediction

• URL: http://arxiv.org/abs/2608.10878v2
• Date: Wed, 19 Aug 2026 03:16:50 GMT
• Title: X2-Turn: Frame-Synchronous Dual-Head Modeling for Joint Streaming ASR and Turn State Prediction
• Abstract summary: X2-Turn is a frame-synchronous turn state prediction method via delayed-stream modeling.<n>We introduce a frame-synchronous turn state head that operates in parallel with the ASR head on shared streaming representations.
• Score: 6.446583777614073
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurate and responsive turn-taking is essential for spoken dialogue systems, which must distinguish in real time between user interruptions, backchannels that should be ignored, and the completion of an utterance. Prior modular approaches typically optimize turn state prediction at the utterance or fixed-chunk level, creating a mismatch with the continuous turn state estimate, and often depend on an auxiliary ASR model, which limits responsiveness and increases overall system complexity. Therefore, we present X2-Turn, a frame-synchronous turn state prediction method via delayed-stream modeling. Specifically, building on the pretrained Voxtral Realtime model, we introduce a frame-synchronous turn state head that operates in parallel with the ASR head on shared streaming representations, jointly predicting ASR tokens and fine-grained turn states at the frame level. We evaluate our method on the bilingual Chinese-English Easy-Turn test sets, and the results demonstrate its effectiveness in achieving accurate turn-taking detection while maintaining low latency.

Related papers

• StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models [84.17128030384099]
  StreamPI is a streaming multimodal temporal modeling framework.<n>It equips single-frame VLA with temporal reasoning capability without introducing any additional parameters.<n> Experiments on real-robot tasks spanning memory-dependent and precise perception scenarios, as well as the simulation benchmark LIBERO, demonstrate that StreamPI outperforms pi0.5 across diverse tasks.
  arXiv  Detail & Related papers  (2026-08-26T17:33:19Z)
• COMET: Contrastive Motion-Enhanced Temporal Reasoning for Video Multimodal Large Language Models [15.39037309359101]
  Video multimodal large language models have advanced significantly, yet fine-grained motion-temporal understanding remains fragile.<n>We propose COMET, a temporally grounded framework that systematically strengthens video MLLMs through explicit temporal representation.<n>For optimization, COMET combines temporal prior distillation with a forward-reverse TC-GRPO stage that turns temporal order into a direct learning signal.
  arXiv  Detail & Related papers  (2026-08-21T12:28:36Z)
• Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering [87.43777061308658]
  We analyze predictive behavior in full spoken language models (FDSLMs)<n>We show that FDSLMs dynamically modulate their internal focus between two states: a generative state aligned with model output generation and a perceptive state aligned incoming user input.<n>During user interruptions, the model remains transiently biased toward the generative state before transitioning into the perceptive state, causing it miss the beginning of incoming input.
  arXiv  Detail & Related papers  (2026-06-09T19:08:07Z)
• Synchronization and Turn-Taking in Full-Duplex Speech Dialogue Models [3.5946669116828134]
  Full spoken dialogue models can speak simultaneously closer to human conversation than turn-based systems.<n>We study such models coordinate their internal interaction using neural coupling in human communication.<n>We find strong representational synchronization under no noise conditions, peaking near zero lag and degrading noise.
  arXiv  Detail & Related papers  (2026-05-19T18:11:03Z)
• LeapTS: Rethinking Time Series Forecasting as Adaptive Multi-Horizon Scheduling [74.94985663101906]
  We present LeapTS, a novel framework that reformulates time series forecasting as a dynamic scheduling process over the prediction horizon.<n>We show that LeapTS improves overall forecasting performance by at least 7.4% while achieving a 2.6$times$ to 5.3$times$ inference speedup over representative Transformer-based models.
  arXiv  Detail & Related papers  (2026-05-11T09:54:02Z)
• An ALE-Consistent Graph Neural Operator-Transformer Framework for Fluid-Structure Interaction [0.0]
  We propose an arbitrary Lagrangian-Eulerian (ALE)-consistent machine learning framework for prediction on deforming unstructured meshes.<n>The framework is on the benchmark problem of a flexible beam vibration in the wake of a cylinder.
  arXiv  Detail & Related papers  (2026-05-01T03:36:18Z)
• JAL-Turn: Joint Acoustic-Linguistic Modeling for Real-Time and Robust Turn-Taking Detection in Full-Duplex Spoken Dialogue Systems [3.0789720666469407]
  JAL-Turn is a speech-only turn-taking framework that runs fully in parallel with speech recognition.<n>JAL-Turn consistently outperforms strong state-of-the-art baselines in detection accuracy while maintaining superior real-time performance.
  arXiv  Detail & Related papers  (2026-03-27T15:25:38Z)
• Parallel BiLSTM-Transformer networks for forecasting chaotic dynamics [24.960864709838436]
  This study proposes a parallel predictive framework integrating Transformer and Bidirectional Long Short-Term Memory networks.<n>The proposed hybrid model employs a dual-branch architecture, where the Transformer branch mainly captures long-range dependencies.<n>The results consistently indicate that the proposed hybrid framework outperforms both single-branch architectures across tasks.
  arXiv  Detail & Related papers  (2025-10-27T16:17:10Z)
• ReCoM: Realistic Co-Speech Motion Generation with Recurrent Embedded Transformer [58.49950218437718]
  We present ReCoM, an efficient framework for generating high-fidelity and generalizable human body motions synchronized with speech.<n>The core innovation lies in the Recurrent Embedded Transformer (RET), which integrates Dynamic Embedding Regularization (DER) into a Vision Transformer (ViT) core architecture.<n>To enhance model robustness, we incorporate the proposed DER strategy, which equips the model with dual capabilities of noise resistance and cross-domain generalization.
  arXiv  Detail & Related papers  (2025-03-27T16:39:40Z)
• Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities [93.09944267871163]
  FullDuplexBench is a benchmark that systematically evaluates key interactive behaviors.<n>By releasing our benchmark code we aim to advance spoken dialogue modeling and the development of more natural and engaging SDMs.
  arXiv  Detail & Related papers  (2025-03-06T18:59:16Z)
• Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Model [63.336123527432136]
  We introduce Bench2Drive-R, a generative framework that enables reactive closed-loop evaluation.<n>Unlike existing video generative models for autonomous driving, the proposed designs are tailored for interactive simulation.<n>We compare the generation quality of Bench2Drive-R with existing generative models and achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-12-11T06:35:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.