Beyond Fresh Starts: Stateful Inference for Streaming ASR in Conversational Voice Agents
Abstract Overview
This paper analyzes how state handling in low-latency streaming ASR affects conversational voice agents, focusing on turn boundaries, long silences, and short backchannel turns. Using two cache-aware FastConformer-RNNT streaming models on CallHome and Switchboard, the authors compare standard per-turn state resets, continuous streaming, and two proposed inference-time state-management strategies. The proposed methods preserve cross-utterance context while avoiding the harmful effects of silence and backchannel contamination in the model state. Across evaluations at 160 ms latency and related settings, the study finds that onset errors are a key weakness of fresh-start decoding and that selective state preservation improves recognition.
Novelty
The main novelty is an inference-time treatment of streaming ASR state as a state-management problem for conversational agents, rather than changing training procedures or adding explicit text-context mechanisms. The paper introduces two simple strategies—State Carry-over and State Rollback—that preserve useful prior context across turns, with rollback selectively discarding context from very short turns such as backchannels.
Results
The best method, State Rollback, achieves the lowest WER across both evaluated models and datasets, with average relative first-word/onset WER reductions of about 15–21% over fresh-start baselines. Improvements over fresh-start decoding are reported as statistically significant across all four model-dataset combinations, and the gains persist across sub-250 ms latencies and when using VAD-derived turn boundaries instead of ground-truth timing.
Key Points
- Resetting ASR state at every turn avoids silence accumulation but causes a substantial penalty on the first words of the next utterance because prior context is discarded.
- Continuous streaming without state control is vulnerable to degradation from long pauses and short backchannel turns, which can pollute the limited acoustic memory.
- Selective state preservation, especially State Rollback, consistently provides the strongest trade-off between retaining useful cross-turn context and avoiding harmful conversational noise.