AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing

• URL: http://arxiv.org/abs/2609.08936v1
• Date: Tue, 08 Sep 2026 15:59:16 GMT
• Title: AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing
• Abstract summary: We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context.<n>We construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families.
• Score: 74.97664372105693
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.

Related papers

• Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing [63.573256490583724]
  Audio- Omni is the first end-to-end framework to unify generation and editing across general sound, music, and speech domains.<n>Our architecture synergizes a frozen Multimodal Large Language Model for high-level reasoning with a trainable Diffusion Transformer for high-fidelity synthesis.<n>AudioEdit is a large-scale dataset comprising over one million meticulously curated editing pairs.
  arXiv  Detail & Related papers  (2026-04-12T16:08:20Z)
• UniTalking: A Unified Audio-Video Framework for Talking Portrait Generation [9.86219488808079]
  We introduce UniTalking, a unified, end-to-end diffusion framework for generating high-fidelity speech and lip-synchronized video.<n>By leveraging powerful priors from a pre-trained video generation model, our framework ensures state-of-the-art visual fidelity while enabling efficient training.
  arXiv  Detail & Related papers  (2026-03-02T03:46:53Z)
• AzeroS: Extending LLM to Speech with Self-Generated Instruction-Free Tuning [49.68129589035101]
  We introduce AZeroS (Auden Zero-instruction-tuned Speech-LLM), which is trained on speech-text pairs derived from publicly available corpora.<n>AZeroS achieves state-of-the-art performance on both semantic and paralinguistic benchmarks.
  arXiv  Detail & Related papers  (2025-12-31T04:05:04Z)
• Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representation [43.25237304356213]
  We introduce a novel framework that unifies speech understanding, generation, and editing.<n>The core of our unified model is a unified continuous audio tokenizer MingTok-Audio, the first continuous tokenizer to effectively integrate semantic and acoustic features.<n>Based on this unified continuous audio tokenizer, we developed the speech language model Ming-UniAudio, which achieved a balance between generation and understanding capabilities.
  arXiv  Detail & Related papers  (2025-10-26T17:55:34Z)
• UniVerse-1: Unified Audio-Video Generation via Stitching of Experts [59.38012380516272]
  We introduce UniVerse-1, a unified, Veo-3-like model capable of simultaneously generating coordinated audio and video.<n>To enhance training efficiency, we bypass training from scratch and instead employ a stitching of experts (SoE) technique.
  arXiv  Detail & Related papers  (2025-09-07T17:55:03Z)
• Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction [9.101978573666546]
  Baichuan-Audio is an end-to-end audio large language model that seamlessly integrates audio understanding and generation.<n>It features a text-guided aligned speech generation mechanism, enabling real-time speech interaction with both comprehension and generation capabilities.
  arXiv  Detail & Related papers  (2025-02-24T15:16:34Z)
• Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction [110.38946048535033]
  This paper introduces Step-Audio, the first production-ready open-source solution for speech recognition.<n>Key contributions include: 1) a unified speech-text multi-modal model that achieves unified understanding and generation, with the Step-Audio-Chat version open-sourced; 2) a generative speech data engine that establishes an affordable voice cloning framework and produces the open-sourced lightweight Step-Audio-TTS-3B model through distillation; 3) an instruction-driven fine control system enabling dynamic adjustments across dialects, emotions, singing, and RAP; 4) an enhanced cognitive architecture augmented with tool calling and role-playing abilities to manage complex
  arXiv  Detail & Related papers  (2025-02-17T15:58:56Z)
• FINALLY: fast and universal speech enhancement with studio-like quality [7.207284147264852]
  We address the challenge of speech enhancement in real-world recordings, which often contain various forms of distortion. We study various feature extractors for perceptual loss to facilitate the stability of adversarial training. We integrate WavLM-based perceptual loss into MS-STFT adversarial training pipeline, creating an effective and stable training procedure for the speech enhancement model.
  arXiv  Detail & Related papers  (2024-10-08T11:16:03Z)
• uSee: Unified Speech Enhancement and Editing with Conditional Diffusion Models [57.71199494492223]
  We propose a Unified Speech Enhancement and Editing (uSee) model with conditional diffusion models to handle various tasks at the same time in a generative manner. Our experiments show that our proposed uSee model can achieve superior performance in both speech denoising and dereverberation compared to other related generative speech enhancement models.
  arXiv  Detail & Related papers  (2023-10-02T04:36:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.