Low-Latency Real-Time Audio Game Commentary System via LLM-Based Parallel Text Generation

• URL: http://arxiv.org/abs/2606.13322v1
• Date: Thu, 11 Jun 2026 13:15:13 GMT
• Title: Low-Latency Real-Time Audio Game Commentary System via LLM-Based Parallel Text Generation
• Abstract summary: We present a low-latency real-time audio game commentary system that generates spoken commentary directly from live gameplay video.<n>Conventional pipelines capture frames, generate text, and synthesize speech sequentially for each utterance, and do not request the next generation until speech playback has completed.<n>Our system runs text generation in parallel with speech playback and buffers multiple candidate utterances ahead of time, enabling immediate synthesis at playback boundaries.
• Score: 59.178597408747095
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a low-latency real-time audio game commentary system that generates spoken commentary directly from live gameplay video. In this end-to-end setting, a key bottleneck is accumulated waiting time; conventional pipelines capture frames, generate text, and synthesize speech sequentially for each utterance, and do not request the next generation until speech playback has completed. This strict sequentiality causes long and unnatural silence between utterances. To address this latency bottleneck, our system runs text generation in parallel with speech playback and buffers multiple candidate utterances ahead of time, enabling immediate synthesis at playback boundaries. Experiments on fast-paced game videos show that our parallel design reduces the mean inter-utterance silence from 9.6 seconds to 0.3 seconds compared to sequential baselines. It also improves similarity to professional speaking--silence timing patterns by over 40 %, and a user study with 120 experienced game players confirms significantly improved perceived speaking rhythm. Our demo video is available at: https://youtu.be/pmrRUlvav8M.

Related papers

• Vorch-Streamer: Extending Human Audio-Visual Generation to Real-Time Long-Form Streaming [24.73791931334844]
  Vorch-Streamer is a post-training framework that enables real-time long-form Text-to-Audio-Video (T2AV) streaming.<n>We construct a synthetic corpus of 80K avatar clips spanning 12-21 seconds and first train a causal generator with mixed Teacher Forcing and Diffusion Forcing.<n>With bounded causal context and four-step denoising, Vorch-Streamer jointly generates audio and video from text at 27.12 FPS, exceeding the 24-FPS real-time playback rate.
  arXiv  Detail & Related papers  (2026-08-06T07:00:37Z)
• Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding [69.296913137409]
  We introduce a novel paradigm for online video understanding: Streaming Video-Language Synchrony (SVLS)<n>We present LyraV, a live streaming assistant built upon a hierarchical control framework with two core innovations.<n>First, the Frame-Driven Transition Controller (FDTC), makes high-level semantic decisions on when to continue speaking, start a new response, or stay silent.<n>Second, the Streaming Token Pacer (SToP), a plug-and-play lightweight predictive module, dynamically adapts the language generation rate to match the pace of the visual content.
  arXiv  Detail & Related papers  (2026-06-05T07:29:20Z)
• StreamChar: Long-Horizon Streaming Character Audio-Video Generation with Decoupled Orchestration [16.23723735702324]
  We present StreamChar, a streaming framework that separates long-horizon orchestration from short-window audio-video denoising.<n>Experiments on short-clip and long-horizon protocols show that StreamChar runs in real time on a single H100 GPU.
  arXiv  Detail & Related papers  (2026-05-25T10:04:52Z)
• StreamingTalker: Audio-driven 3D Facial Animation with Autoregressive Diffusion Model [73.30619724574642]
  Speech-driven 3D facial animation aims to generate realistic and synchronized facial motions driven by speech inputs.<n>Recent methods have employed audio-conditioned diffusion models for 3D facial animation.<n>We propose a novel autoregressive diffusion model that processes audio in a streaming manner.
  arXiv  Detail & Related papers  (2025-11-18T07:55:16Z)
• STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Models [131.90117151306993]
  Spoken Language Models (SLMs) are designed to take speech inputs and produce spoken responses.<n>Current SLMs lack the ability to perform an internal, unspoken thinking process before responding.<n>We propose Stitch, a novel generation method that alternates between the generation of unspoken reasoning chunks and spoken response chunks.
  arXiv  Detail & Related papers  (2025-07-21T08:30:03Z)
• StreamMel: Real-Time Zero-shot Text-to-Speech via Interleaved Continuous Autoregressive Modeling [50.537794606598254]
  StreamMel is a pioneering single-stage streaming TTS framework that models continuous mel-spectrograms.<n>It enables low-latency, autoregressive synthesis while preserving high speaker similarity and naturalness.<n>It even achieves performance comparable to offline systems while supporting efficient real-time generation.
  arXiv  Detail & Related papers  (2025-06-14T16:53:39Z)
• IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities [55.11130688075417]
  We introduce IntrinsicVoic,e an LLM designed with intrinsic real-time voice interaction capabilities. Our novelty architecture, GroupFormer, can reduce speech sequences to lengths comparable to text sequences. We construct a multi-turn speech-to-speech dialogue dataset named method-500k which includes nearly 500k turns of speech-to-speech dialogues.
  arXiv  Detail & Related papers  (2024-10-09T05:04:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.