Modeling Overlapped Speech with Shuffles

• URL: http://arxiv.org/abs/2603.17769v1
• Date: Wed, 18 Mar 2026 14:28:58 GMT
• Title: Modeling Overlapped Speech with Shuffles
• Abstract summary: We propose to model parallel streams of data, such as overlapped speech, using shuffles.<n>We show how the shuffle product and partial order finite-state automata (FSAs) can be used for alignment and speaker-attributed transcription of overlapped speech.
• Score: 57.278869801844316
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose to model parallel streams of data, such as overlapped speech, using shuffles. Specifically, this paper shows how the shuffle product and partial order finite-state automata (FSAs) can be used for alignment and speaker-attributed transcription of overlapped speech. We train using the total score on these FSAs as a loss function, marginalizing over all possible serializations of overlapping sequences at subword, word, and phrase levels. To reduce graph size, we impose temporal constraints by constructing partial order FSAs. We address speaker attribution by modeling (token, speaker) tuples directly. Viterbi alignment through the shuffle product FSA directly enables one-pass alignment. We evaluate performance on synthetic LibriSpeech overlaps. To our knowledge, this is the first algorithm that enables single-pass alignment of multi-talker recordings. All algorithms are implemented using k2 / Icefall.

Related papers

• PairAlign: A Framework for Sequence Tokenization via Self-Alignment with Applications to Audio Tokenization [8.126475706520063]
  PairAlign is a framework for compact audio tokenization through sequence-level self-alignment.<n>It learns compact, non-degenerate sequences with broad vocabulary usage.<n>It preserves edit-distance search while reducing archive token count by 55%.
  arXiv  Detail & Related papers  (2026-05-07T17:11:22Z)
• Zonkey: A Hierarchical Diffusion Language Model with Differentiable Tokenization and Probabilistic Attention [0.0]
  Zonkey is a hierarchical diffusion model that addresses limitations through a fully trainable pipeline from raw characters to document-level representations.<n>At its core is a differentiable tokenizer that learns probabilistic beginning-of-sequence (BOS) decisions.<n>Zonkey generates coherent, variable-length text from noise, demonstrating emergent hierarchies.
  arXiv  Detail & Related papers  (2026-01-29T14:17:37Z)
• DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusion [28.204167153140506]
  Speech tokenizers serve as the cornerstone of discrete Speech Large Language Models.<n>We propose DSA-Tokenizer, which explicitly disentangles speech into discrete semantic and acoustic tokens.
  arXiv  Detail & Related papers  (2026-01-14T07:22:24Z)
• Decoding Order Matters in Autoregressive Speech Synthesis [11.222948749269515]
  Autoregressive speech synthesis often adopts a left-to-right order, yet generation order is a modelling choice.<n>We investigate decoding order through masked diffusion framework, which progressively unmasks positions.<n>We show that randomness in decoding order affects speech quality.
  arXiv  Detail & Related papers  (2026-01-13T11:21:36Z)
• Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesis [64.12708207721276]
  We introduce a novel pseudo-autoregressive (PAR) language modeling approach that unifies AR and NAR modeling.<n>Building on PAR, we propose PALLE, a two-stage TTS system that leverages PAR for initial generation followed by NAR refinement.<n>Experiments demonstrate that PALLE, trained on LibriTTS, outperforms state-of-the-art systems trained on large-scale data.
  arXiv  Detail & Related papers  (2025-04-14T16:03:21Z)
• Hierarchical Phrase-based Sequence-to-Sequence Learning [94.10257313923478]
  We describe a neural transducer that maintains the flexibility of standard sequence-to-sequence (seq2seq) models while incorporating hierarchical phrases as a source of inductive bias during training and as explicit constraints during inference. Our approach trains two models: a discriminative derivation based on a bracketing grammar whose tree hierarchically aligns source and target phrases, and a neural seq2seq model that learns to translate the aligned phrases one-by-one.
  arXiv  Detail & Related papers  (2022-11-15T05:22:40Z)
• token2vec: A Joint Self-Supervised Pre-training Framework Using Unpaired Speech and Text [65.04385919645395]
  token2vec is a novel joint pre-training framework for unpaired speech and text based on discrete representations of speech. Experiments show that token2vec is significantly superior to various speech-only pre-training baselines, with up to 17.7% relative WER reduction.
  arXiv  Detail & Related papers  (2022-10-30T06:38:19Z)
• Speaker Embedding-aware Neural Diarization: a Novel Framework for Overlapped Speech Diarization in the Meeting Scenario [51.5031673695118]
  We reformulate overlapped speech diarization as a single-label prediction problem. We propose the speaker embedding-aware neural diarization (SEND) system.
  arXiv  Detail & Related papers  (2022-03-18T06:40:39Z)
• Continuous speech separation: dataset and analysis [52.10378896407332]
  In natural conversations, a speech signal is continuous, containing both overlapped and overlap-free components. This paper describes a dataset and protocols for evaluating continuous speech separation algorithms.
  arXiv  Detail & Related papers  (2020-01-30T18:01:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.