An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation

• URL: http://arxiv.org/abs/2607.02119v1
• Date: Thu, 02 Jul 2026 12:55:11 GMT
• Title: An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation
• Abstract summary: We present a vLLM-based inference pipeline for unified speech understanding and generation.<n>We overcome the shared intuition-Free Guidance (CFG) halves throughput.<n>By co-scheduling paired conditional and unconditional requests within a continuous batch, our CFG implementation sustains 80% of non-CFG throughput.
• Score: 63.50393485685279
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While Large Multimodal Models excel in comprehension, high-throughput inference engines lack native support for multimodal generation. This is severe in Speech Language Models, where generating multi-layered audio tokens via decoupled AR+NAR or synchronous Multi-Token Prediction (MTP) with delay-pattern interleaving conflicts with standard single-stream loops. We present a vLLM-based inference pipeline for unified speech understanding and generation. We extend autoregressive decoding to natively execute delay-pattern de-interleaving and coordinated multi-stream sampling, integrating an on-GPU acoustic decoder for end-to-end waveform synthesis. Crucially, we overcome the shared intuition that Classifier-Free Guidance (CFG) halves throughput. By co-scheduling paired conditional and unconditional requests within a continuous batch, our CFG implementation sustains 80% of non-CFG throughput, absorbing dual-request and logit merging overheads. We open-source our framework.

Related papers

• Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models [65.49950267695267]
  We introduce Omni2LoRA, a framework for efficient parametric memory compression via coherence-preserving context distillation.<n>By converting multimodal memory into a fixed-budget, reusable parameter state, our method drives answer-time multimodal-token load to zero.
  arXiv  Detail & Related papers  (2026-08-10T07:49:10Z)
• DiffusionGemma Technical Report [72.10570774996219]
  We introduce DiffusionGemma, a language model that uses discrete diffusion to generate text at exceptionally high speed.<n>Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel.<n>Our compute-efficient two-stage training pipeline uses fewer than 10% of the starting AR model's total training token budget.
  arXiv  Detail & Related papers  (2026-07-31T16:11:46Z)
• ReGen: Hierarchical Multi-Prompt Representation Generation for Efficient Waveform Diffusion Models [22.63266813055847]
  We propose ReGen that jointly estimates multiple vector fields for both representations data within a single diffusion model.<n>We also present ReGenVoice, a latent diffusion model (LDM)-based text-to-speech model that achieves strong speech intelligibility.
  arXiv  Detail & Related papers  (2026-07-10T06:44:04Z)
• Stream2LLM: Overlap Context Streaming and Prefill for Reduced Time-to-First-Token (TTFT) [2.8350851063316873]
  Streaming context incrementally--overlapping retrieval with inference--can mitigate this latency.<n>We present Stream2LLM, a streaming-aware LLM serving system for concurrent prefill-decode disaggregated deployments.
  arXiv  Detail & Related papers  (2026-03-29T06:49:12Z)
• OmniForcing: Unleashing Real-time Joint Audio-Visual Generation [51.031726911410594]
  Recent joint audio-visual diffusion models achieve remarkable generation quality but suffer from high latency.<n>We propose OmniForcing, the first framework to distill an offline, dual-stream bidirectional diffusion model into a high-fidelity streaming autore generator.
  arXiv  Detail & Related papers  (2026-03-12T08:17:36Z)
• Diffusion-Driven Semantic Communication for Generative Models with Bandwidth Constraints [66.63250537475973]
  This paper introduces a diffusion-driven semantic communication framework with advanced VAE-based compression for bandwidth-constrained generative model.<n>Our experimental results demonstrate significant improvements in pixel-level metrics like peak signal to noise ratio (PSNR) and semantic metrics like learned perceptual image patch similarity (LPIPS)
  arXiv  Detail & Related papers  (2024-07-26T02:34:25Z)
• Alto: Orchestrating Distributed Compound AI Systems with Nested Ancestry [18.671852079446605]
  Compound AI applications chain together subcomponents such as generative language models, document retrievers, and embedding models.<n>Applying traditional systems optimizations such as parallelism and pipelining in compound AI systems is difficult because each component has different constraints.<n>We present Alto, a framework that automatically optimize execution of compound AI queries through streaming and parallelism.
  arXiv  Detail & Related papers  (2024-03-07T08:30:26Z)
• StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation [52.56469577812338]
  We introduce StreamDiffusion, a real-time diffusion pipeline for interactive image generation.<n>Existing diffusion models are adept at creating images from text or image prompts, yet they often fall short in real-time interaction.<n>We present a novel approach that transforms the original sequential denoising into the denoising process.
  arXiv  Detail & Related papers  (2023-12-19T18:18:33Z)
• Turbocharge Speech Understanding with Pilot Inference [0.9699101045941684]
  This paper sets to accelerate modern speech understanding on resource-constrained edge devices. It takes a hybrid approach: to speed up on-device execution; to offload inputs that are beyond the device's capacity. Our prototype, called PASU, is tested on Arm platforms with 6 - 8 cores: it delivers SOTA accuracy; it reduces the end-to-end latency by 2x and reduces the offloading needs by 2x.
  arXiv  Detail & Related papers  (2023-11-22T17:14:18Z)
• Speech Synthesis By Unrolling Diffusion Process using Neural Network Layers [3.2634122554914002]
  UDPNet is a novel architecture designed to accelerate the reverse diffusion process in speech synthesis.<n>We show that UDPNet consistently outperforms state-of-the-art methods in both quality and efficiency.<n>These results position UDPNet as a robust solution for real-time speech synthesis applications.
  arXiv  Detail & Related papers  (2023-09-18T10:35:27Z)
• FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis [90.3069686272524]
  This paper proposes FastDiff, a fast conditional diffusion model for high-quality speech synthesis. FastDiff employs a stack of time-aware location-variable convolutions of diverse receptive field patterns to efficiently model long-term time dependencies. Based on FastDiff, we design an end-to-end text-to-speech synthesizer, FastDiff-TTS, which generates high-fidelity speech waveforms.
  arXiv  Detail & Related papers  (2022-04-21T07:49:09Z)
• Raw Waveform Encoder with Multi-Scale Globally Attentive Locally Recurrent Networks for End-to-End Speech Recognition [45.858039215825656]
  We propose a new encoder that adopts globally attentive locally recurrent (GALR) networks and directly takes raw waveform as input. Experiments are conducted on a benchmark dataset AISHELL-2 and two large-scale Mandarin speech corpus of 5,000 hours and 21,000 hours.
  arXiv  Detail & Related papers  (2021-06-08T12:12:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.