TRADE: Transducer-Augmented Decoder for Speech LLM

• URL: http://arxiv.org/abs/2606.08486v1
• Date: Sun, 07 Jun 2026 07:15:34 GMT
• Title: TRADE: Transducer-Augmented Decoder for Speech LLM
• Abstract summary: Speech Large Language Models (Speech LLMs) lack a principled mechanism for streaming inference.<n>We propose TRADE TRansducer-Augmented DEcoder, which augments a multimodal LLM with a transducer branch that shares the audio encoder.<n>Three design choices make the system accurate, streamable, and long-form capable.
• Score: 52.69726809996728
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Speech Large Language Models (Speech LLMs) lack a principled mechanism for streaming inference: their label-synchronous generation has no acoustic-frame alignment, making real-time decoding and end-of-utterance detection difficult. We propose TRADE TRansducer-Augmented DEcoder, which augments a multimodal LLM with a transducer branch that shares the audio encoder and uses the LLM's hidden states directly as the prediction network -- coupling frame-synchronous acoustic alignment with the LLM's linguistic reasoning. Three design choices make the system accurate, streamable, and long-form capable: (1)Tightly coupled dual vocabularies -- a compact transducer vocabulary derived from the LLM vocabulary, enabling zero-cost score fusion; (2)Chunk-synchronized streaming training with gradient stopping, eliminating the train-inference mismatch at offline-equivalent memory cost; and (3)Localized Decoder Audio Attention (LDAA), a causal sliding window that caps KV-cache memory independently of utterance length. A single TRADE checkpoint supports offline and streaming decoding across a continuous range of latency operating points. TRADE achieves 6.71% average WER on the Open ASR Leaderboard, while the streaming recognition with 960ms chunk size reaches 8.40% from the same checkpoint. On long-form speech, it obtains 3.64% WER on TED-LIUM and 10.88% on Earnings-22 without external segmentation. TRADE provides sentence-end punctuation timestamps that, when combined with acoustic voice activity detection (VAD), improve end-of-utterance detection by +0.03 F_1 over acoustic VAD alone.

Related papers

• Zero-Shot Temporal Localisation of Audio Deepfakes in Multi-Speaker Conversations [4.152774504513197]
  Training-free five-stage pipeline wraps a frozen binary detector and adds segment-level output with no retraining.<n>On 180 constructed multi-speaker conversations from ASVspoof 5, the system attains temporal intersection-over-union 0.90, temporal detection rate 0.95, and MS-DCF 0.26 with a strong backbone.<n>Its false-alarm rate on genuine speech is below 6%, falling under 2% on genuine real multi-speaker dialogue (AMI)<n>Under an identical pipeline, a trained localiser improves temporal IoU by only about 0.04, bounding the cost of forgoing supervision.
  arXiv  Detail & Related papers  (2026-09-09T11:27:26Z)
• Memory Efficient Audio Synthesis with Decoupled Temporal Depth Diffusion Transformers [11.102698205402675]
  Siri Expressive Voices synthesizes rich, conversational speech in real time and entirely on device.<n>A detokenizer converts semantic audio tokens emitted by the foundation model into high-fidelity audio within the tight compute and memory budget of the Apple Matrix Coprocessor (AMX)<n>A single reusable depth decoder with Diffusion Transformer (DiT)-style stage conditioning generates all RVQ levels autoregressively, replacing the dedicated per-level decoders of prior multi-decoder architectures.
  arXiv  Detail & Related papers  (2026-07-26T19:20:01Z)
• UnityShots: Memory-Driven Multi-Shot Audio-Video Generation with Boundary-Aware Gating [60.20680928161925]
  We present UnityShots, a memory-driven multi-shot audio-video generation system built on LTX-2.3.<n>The video stream maintains two fixed-size slots, a long-term memory (LTM) slot anchored to the opening shot and a short-term memory (STM) slot holding the immediately preceding tail.<n>The audio stream injects a reference speaker token at every shot to preserve vocal timbre without a sliding audio bank.
  arXiv  Detail & Related papers  (2026-06-19T18:06:15Z)
• Diffusion Large Language Models for Visual Speech Recognition [56.81307584718608]
  We propose the first Diffusion Large Language Model (DLLM)-based Visual Speech Recognition (VSR) framework.<n>DLLM-VSR commits high-confidence positions early and uses the committed tokens as bidirectional context to refine ambiguous ones.<n>We develop length-guided candidate decoding, which uses video duration to construct plausible transcript-length hypotheses.<n>The proposed method achieves a state-of-the-art WER of 19.5% on LRS3 using only its labeled training data.
  arXiv  Detail & Related papers  (2026-05-27T13:22:08Z)
• WISV: Wireless-Informed Semantic Verification for Distributed Speculative Decoding in Device-Edge LLM Inference [56.297697169678095]
  WISV (Wireless-Informed Semantic Verification) is a novel distributed speculative decoding framework.<n>We show that WISV achieves up to a 60.8% increase in accepted length, a 37.3% reduction in interaction rounds, and a 31.4% improvement in end-to-end latency.<n>We validate WISV on a hardware testbed comprising an NVIDIA Jetson AGX Orin and an A40-equipped server.
  arXiv  Detail & Related papers  (2026-04-20T01:29:56Z)
• Temporal Contrastive Decoding: A Training-Free Method for Large Audio-Language Models [56.91801348360746]
  Large audio-language models (LALMs) generalize across speech, sound, and music.<n> unified decoders can exhibit a emphtemporal smoothing bias<n>We propose emphTemporal Contrastive Decoding (TCD), a training-free decoding method for unified LALMs.
  arXiv  Detail & Related papers  (2026-04-16T02:30:41Z)
• Who Spoke What When? Evaluating Spoken Language Models for Conversational ASR with Semantic and Overlap-Aware Metrics [99.89493037369071]
  Conversational automatic speech recognition remains challenging due to overlapping speech, far-field noise, and varying speaker counts.<n>Recent LLM-based systems perform well on single-speaker benchmarks, but their robustness in multi-speaker settings is unclear.<n>We systematically compare LLM-based and modular approaches along four axes: overlap, semantic fidelity, speaker count, and single- versus multi-channel input.
  arXiv  Detail & Related papers  (2026-03-24T02:01:21Z)
• MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokens [40.95973318669374]
  We introduce an efficient multimodal speech LLM framework that minimizes token length while preserving essential linguistic content.<n>Our approach achieves state-of-the-art performance with a WER of 0.72% while using only 3.5 tokens per second.
  arXiv  Detail & Related papers  (2025-03-14T11:31:30Z)
• Large Language Models are Strong Audio-Visual Speech Recognition Learners [53.142635674428874]
  Multimodal large language models (MLLMs) have recently become a focal point of research due to their formidable multimodal understanding capabilities.<n>We propose Llama-AVSR, a new MLLM with strong audio-visual speech recognition capabilities.<n>We evaluate our proposed approach on LRS3, the largest public AVSR benchmark, and we achieve new state-of-the-art results for the tasks of ASR and AVSR with a WER of 0.79% and 0.77%, respectively.
  arXiv  Detail & Related papers  (2024-09-18T21:17:27Z)
• Streaming End-to-End Multilingual Speech Recognition with Joint Language Identification [14.197869575012925]
  We propose to modify the structure of the cascaded-encoder-based recurrent neural network transducer (RNN-T) model by integrating a per-frame language identifier (LID) predictor. RNN-T with cascaded encoders can achieve streaming ASR with low latency using first-pass decoding with no right-context, and achieve lower word error rates (WERs) using second-pass decoding with longer right-context. Experimental results on a voice search dataset with 9 language locales shows that the proposed method achieves an average of 96.2% LID prediction accuracy and the same second-pass WER
  arXiv  Detail & Related papers  (2022-09-13T15:10:41Z)
• Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selection [15.525314212209562]
  We propose three latency reduction techniques for chunk-based incremental inference. We show that our approach is also applicable to low-latency speech translation.
  arXiv  Detail & Related papers  (2020-05-22T13:42:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.