A Hierarchical End-of-Turn Model with Primary Speaker Segmentation for Real-Time Conversational AI

• URL: http://arxiv.org/abs/2603.13379v1
• Date: Tue, 10 Mar 2026 21:17:17 GMT
• Title: A Hierarchical End-of-Turn Model with Primary Speaker Segmentation for Real-Time Conversational AI
• Abstract summary: We present a real-time front-end for voice-based conversational AI to enable natural turn-taking in two-speaker scenarios.<n>To operate robustly in multi-speaker environments, the system continuously identifies and tracks the primary user.
• Score: 3.995809098639418
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a real-time front-end for voice-based conversational AI to enable natural turn-taking in two-speaker scenarios by combining primary speaker segmentation with hierarchical End-of-Turn (EOT) detection. To operate robustly in multi-speaker environments, the system continuously identifies and tracks the primary user, ensuring that downstream EOT decisions are not confounded by background conversations. The tracked activity segments are fed to a hierarchical, causal EOT model that predicts the immediate conversational state by independently analyzing per-speaker speech features from both the primary speaker and the bot. Simultaneously, the model anticipates near-future states ($t{+}10/20/30$\,ms) through probabilistic predictions that are aware of the conversation partner's speech. Task-specific knowledge distillation compresses wav2vec~2.0 representations (768\,D) into a compact MFCC-based student (32\,D) for efficient deployment. The system achieves 82\% multi-class frame-level F1 and 70.6\% F1 on Backchannel detection, with 69.3\% F1 on a binary Final vs.\ Others task. On an end-to-end turn-detection benchmark, our model reaches 87.7\% recall vs.\ 58.9\% for Smart Turn~v3 while keeping a median detection latency of 36\,ms versus 800--1300\,ms. Despite using only 1.14\,M parameters, the proposed model matches or exceeds transformer-based baselines while substantially reducing latency and memory footprint, making it suitable for edge deployment.

Related papers

• Realtime-Venus: A full-duplex interaction system with asynchronous delegation [59.69745588222872]
  We present Realtime, Realtime Omni audio visual interaction and Realtime-Audio Spoken for spoken interaction.<n>Each model serves as complete conversational, integrating continuous perception, conversational control, and native speech generation.
  arXiv  Detail & Related papers  (2026-09-12T08:56:50Z)
• iPhoneme: Brain-to-Text Communication for ALS Using ConformerXL Decoding [1.529342790344802]
  We present iPhoneme, a brain-to-text communication system for ALS patients.<n>The system combines a deep learning phoneme decoder with a gaze-assisted phoneme input interface.<n>On the interaction side, iPhoneme introduces a chorded gaze-plus-silent-speech paradigm that replaces dwell-time selection.
  arXiv  Detail & Related papers  (2026-04-07T05:08:27Z)
• Towards Robust Overlapping Speech Detection: A Speaker-Aware Progressive Approach Using WavLM [53.17360668423001]
  Overlapping Speech Detection (OSD) aims to identify regions where multiple speakers overlap in a conversation.<n>This work proposes a speaker-aware progressive OSD model that leverages a progressive training strategy to enhance the correlation between subtasks.<n> Experimental results show that the proposed method achieves state-of-the-art performance, with an F1 score of 82.76% on the AMI test set.
  arXiv  Detail & Related papers  (2025-05-29T07:47:48Z)
• An Effective Transformer-based Contextual Model and Temporal Gate Pooling for Speaker Identification [0.0]
  This paper introduces an effective end-to-end speaker identification model applied Transformer-based contextual model. We propose a pooling method, Temporal Gate Pooling, with powerful learning ability for speaker identification. The proposed method has achieved an accuracy of 87.1% with 28.5M parameters, demonstrating comparable precision to wav2vec2 with 317.7M parameters.
  arXiv  Detail & Related papers  (2023-08-22T07:34:07Z)
• LoCoNet: Long-Short Context Network for Active Speaker Detection [18.06037779826666]
  Active Speaker Detection (ASD) aims to identify who is speaking in each frame of a video. We propose LoCoNet, a simple yet effective Long-Short Context Network. LoCoNet achieves state-of-the-art performance on multiple datasets.
  arXiv  Detail & Related papers  (2023-01-19T18:54:43Z)
• Unified End-to-End Speech Recognition and Endpointing for Fast and Efficient Speech Systems [17.160006765475988]
  We propose a method to jointly train the ASR and EP tasks in a single end-to-end (E2E) model. We introduce a "switch" connection, which trains the EP to consume either the audio frames directly or low-level latent representations from the ASR model. This results in a single E2E model that can be used during inference to perform frame filtering at low cost.
  arXiv  Detail & Related papers  (2022-11-01T23:43:15Z)
• Real-time Speaker counting in a cocktail party scenario using Attention-guided Convolutional Neural Network [60.99112031408449]
  We propose a real-time, single-channel attention-guided Convolutional Neural Network (CNN) to estimate the number of active speakers in overlapping speech. The proposed system extracts higher-level information from the speech spectral content using a CNN model. Experiments on simulated overlapping speech using WSJ corpus show that the attention solution is shown to improve the performance by almost 3% absolute over conventional temporal average pooling.
  arXiv  Detail & Related papers  (2021-10-30T19:24:57Z)
• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units [81.53783563025084]
  We propose an offline clustering step to provide aligned target labels for a BERT-like prediction loss. A key ingredient of our approach is applying the prediction loss over the masked regions only. HuBERT shows up to 19% and 13% relative WER reduction on the more challenging dev-other and test-other evaluation subsets.
  arXiv  Detail & Related papers  (2021-06-14T14:14:28Z)
• Sequential End-to-End Intent and Slot Label Classification and Localization [2.1684857243537334]
  end-to-end (e2e) spoken language understanding (SLU) solutions have recently been proposed to decrease latency. We propose a compact e2e SLU architecture for streaming scenarios, where chunks of the speech signal are processed continuously to predict intent and slot values. Results show our model ability to process incoming speech signal, reaching accuracy as high as 98.97 % for CTC and 98.78 % for CTL on single-label classification.
  arXiv  Detail & Related papers  (2021-06-08T19:53:04Z)
• Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling [61.351967629600594]
  This paper proposes an any-to-many location-relative, sequence-to-sequence (seq2seq), non-parallel voice conversion approach. In this approach, we combine a bottle-neck feature extractor (BNE) with a seq2seq synthesis module. Objective and subjective evaluations show that the proposed any-to-many approach has superior voice conversion performance in terms of both naturalness and speaker similarity.
  arXiv  Detail & Related papers  (2020-09-06T13:01:06Z)
• Self-attention encoding and pooling for speaker recognition [16.96341561111918]
  We propose a tandem Self-Attention and Pooling (SAEP) mechanism to obtain a discriminative speaker embedding given non-fixed length speech utterances. SAEP encodes short-term speaker spectral features into speaker embeddings to be used in text-independent speaker verification. We have evaluated this approach on both VoxCeleb1 & 2 datasets.
  arXiv  Detail & Related papers  (2020-08-03T09:31:27Z)
• Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario [51.50631198081903]
  We propose a novel Target-Speaker Voice Activity Detection (TS-VAD) approach. TS-VAD directly predicts an activity of each speaker on each time frame. Experiments on the CHiME-6 unsegmented data show that TS-VAD achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2020-05-14T21:24:56Z)
• A Simple Language Model for Task-Oriented Dialogue [61.84084939472287]
  SimpleTOD is a simple approach to task-oriented dialogue that uses a single, causal language model trained on all sub-tasks recast as a single sequence prediction problem. This allows SimpleTOD to fully leverage transfer learning from pre-trained, open domain, causal language models such as GPT-2.
  arXiv  Detail & Related papers  (2020-05-02T11:09:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.