VibeVoice-ASR-BitNet Technical Report
Abstract Overview
This technical report presents VibeVoice-ASR-BitNet, a compressed version of VibeVoice-ASR designed for real-time automatic speech recognition on edge CPUs. The system uses heterogeneous quantization: the VAE acoustic tokenizer is quantized with a full-pipeline INT8 scheme, while the autoregressive language model uses BitNet-style ternary weights, with additional fused operators and custom SIMD kernels for ARM and x86 CPUs. The authors also apply progressive quantization-aware training to stabilize aggressive compression and preserve recognition quality. In experiments, the model is reduced from 4.62 GB to 1.58 GB and reaches real-time inference with fewer CPU threads than typically required for large ASR models.
Novelty
The distinctive contribution is a stage-specific quantization design that matches different computational bottlenecks within an LLM-based ASR pipeline, rather than applying a single quantization method uniformly. The work also combines this with progressive QAT and custom SIMD kernels in ggml to make ternary/INT8 inference practical on commodity CPUs.
Results
The reported system achieves a 2.9x model-size reduction from 4.62 GB to 1.58 GB and reaches real-time inference (RTF < 1) with as few as 3 CPU threads across tested input lengths. It is reported to be 1.6-2.3x faster than Whisper.cpp at similar model size, while showing only modest degradation relative to the FP16 baseline and competitive multilingual accuracy on several benchmarks.
Key Points
- The paper assigns INT8 quantization to the activation-heavy VAE tokenizer and ternary BitNet-style quantization to the weight-heavy language model, reflecting their different CPU bottlenecks.
- Inference efficiency is supported by operator fusion and custom SIMD kernels for ARM and x86 within the ggml framework, enabling low-thread real-time CPU ASR.
- Across 15 benchmarks, the compressed model retains competitive multilingual ASR performance, though the current implementation is validated on this architecture and supports offline inference only.