End-to-End Learning-based Video Streaming Enhancement Pipeline: A Generative AI Approach

• URL: http://arxiv.org/abs/2512.14185v1
• Date: Tue, 16 Dec 2025 08:28:53 GMT
• Title: End-to-End Learning-based Video Streaming Enhancement Pipeline: A Generative AI Approach
• Abstract summary: ELVIS is an end-to-end architecture that combines server-side encoding optimizations with client-side generative in-painting to remove and reconstruct redundant video data.<n>Current technologies achieve improvements of up to 11 VMAF points over baseline benchmarks.
• Score: 9.347197482893113
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The primary challenge of video streaming is to balance high video quality with smooth playback. Traditional codecs are well tuned for this trade-off, yet their inability to use context means they must encode the entire video data and transmit it to the client. This paper introduces ELVIS (End-to-end Learning-based VIdeo Streaming Enhancement Pipeline), an end-to-end architecture that combines server-side encoding optimizations with client-side generative in-painting to remove and reconstruct redundant video data. Its modular design allows ELVIS to integrate different codecs, inpainting models, and quality metrics, making it adaptable to future innovations. Our results show that current technologies achieve improvements of up to 11 VMAF points over baseline benchmarks, though challenges remain for real-time applications due to computational demands. ELVIS represents a foundational step toward incorporating generative AI into video streaming pipelines, enabling higher quality experiences without increased bandwidth requirements.

Related papers

• Harnessing Streaming Video in the Wild [53.23721420272668]
  Vision-Language Models (VLMs) are increasingly required to process video streams in applications such as video-call assistants, live commentary, and embodied robots.<n>An ideal streaming system should support proactive interaction, long-horizon memory, and real-time processing.<n>ExistingVLMs excel at offline video understanding but fall short in streaming capabilities and lack dedicated infrastructure for streaming deployment.
  arXiv  Detail & Related papers  (2026-06-07T13:00:19Z)
• CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference [4.835489391255295]
  We present CodecSight, a streaming video analytics system built on a key observation that video codecs already extract the temporal and spatial structure of each stream as a byproduct of compression.<n>CodecSight treats this metadata as a low-cost signal to unify runtime optimization across video decoding, visual processing, and LLM prefilling, with transmission reduction as an inherent benefit of operating directly on compressed bitstreams.<n> Experiments show that CodecSight achieves an improvement in throughput of up to 3$times$, and a reduction of up to 87% in GPU compute over state-of-the-art baselines, maintaining competitive accuracy with
  arXiv  Detail & Related papers  (2026-04-07T16:31:45Z)
• EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generation [80.13014959623452]
  EVATok is a framework to produce $textbfE$fficient $textbfV$ideo $textbfA$daptive $textbfTok$enizers.<n>We show that EVATok achieves superior reconstruction and state-of-the-art class-to-video generation on UCF-101.
  arXiv  Detail & Related papers  (2026-03-12T17:59:59Z)
• Exploring Real-Time Super-Resolution: Benchmarking and Fine-Tuning for Streaming Content [1.1926989512978123]
  We introduce a comprehensive dataset - StreamSR - sourced from YouTube.<n>We benchmark 11 state-of-the-art real-time super-resolution models to evaluate their performance for the streaming use-case.<n>We propose EfRLFN, an efficient real-time model that integrates Efficient Channel Attention and a hyperbolic tangent activation function.
  arXiv  Detail & Related papers  (2026-02-11T20:22:06Z)
• SCENE: Semantic-aware Codec Enhancement with Neural Embeddings [9.6489283335586]
  We propose a semantic-aware pre-processing framework that enhances perceptual fidelity.<n>Our method integrates semantic embeddings from a vision-language model into an efficient convolutional architecture.<n>Our results show that semantic-aware pre-processing is an effective approach for enhancing compressed video streams.
  arXiv  Detail & Related papers  (2026-01-29T05:41:28Z)
• Plug-and-Play Versatile Compressed Video Enhancement [57.62582951699999]
  Video compression effectively reduces the size of files, making it possible for real-time cloud computing.<n>However, it comes at the cost of visual quality, challenges the robustness of downstream vision models.<n>We present a versatile-aware enhancement framework that adaptively enhance videos under different compression settings.
  arXiv  Detail & Related papers  (2025-04-21T18:39:31Z)
• Generating, Fast and Slow: Scalable Parallel Video Generation with Video Interface Networks [21.710127132217526]
  We introduce a new paradigm called Video Interface Networks (VINs), which augment DiTs with an abstraction module to enable parallel inference of video chunks.<n>VINs encode global semantics from the noisy input of local chunks and the encoded representations, in turn, guide DiTs in denoising chunks in parallel.<n>Our approach attains state-of-the-art motion smoothness while using 25-40% fewer FLOPs than full generation.
  arXiv  Detail & Related papers  (2025-03-21T21:13:02Z)
• HiTVideo: Hierarchical Tokenizers for Enhancing Text-to-Video Generation with Autoregressive Large Language Models [63.65066762436074]
  HiTVideo aims to address the potential limitations of existing video tokenizers in text-to-video generation tasks.<n>It utilizes a 3D causal VAE with a multi-layer discrete token framework, encoding video content into hierarchically structured codebooks.
  arXiv  Detail & Related papers  (2025-03-14T15:36:39Z)
• STORM: Token-Efficient Long Video Understanding for Multimodal LLMs [116.4479155699528]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• When Video Coding Meets Multimodal Large Language Models: A Unified Paradigm for Video Coding [118.72266141321647]
  Cross-Modality Video Coding (CMVC) is a pioneering approach to explore multimodality representation and video generative models in video coding.<n>During decoding, previously encoded components and video generation models are leveraged to create multiple encoding-decoding modes.<n>Experiments indicate that TT2V achieves effective semantic reconstruction, while IT2V exhibits competitive perceptual consistency.
  arXiv  Detail & Related papers  (2024-08-15T11:36:18Z)
• Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams [78.72965584414368]
  We present Flash-VStream, a video-language model that simulates the memory mechanism of human. Compared to existing models, Flash-VStream achieves significant reductions in latency inference and VRAM consumption. We propose VStream-QA, a novel question answering benchmark specifically designed for online video streaming understanding.
  arXiv  Detail & Related papers  (2024-06-12T11:07:55Z)
• Boosting Neural Representations for Videos with a Conditional Decoder [28.073607937396552]
  Implicit neural representations (INRs) have emerged as a promising approach for video storage and processing. This paper introduces a universal boosting framework for current implicit video representation approaches.
  arXiv  Detail & Related papers  (2024-02-28T08:32:19Z)
• A Coding Framework and Benchmark towards Low-Bitrate Video Understanding [63.05385140193666]
  We propose a traditional-neural mixed coding framework that takes advantage of both traditional codecs and neural networks (NNs) The framework is optimized by ensuring that a transportation-efficient semantic representation of the video is preserved. We build a low-bitrate video understanding benchmark with three downstream tasks on eight datasets, demonstrating the notable superiority of our approach.
  arXiv  Detail & Related papers  (2022-02-06T16:29:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.