Promptus: Can Prompts Streaming Replace Video Streaming with Stable Diffusion

• URL: http://arxiv.org/abs/2405.20032v2
• Date: Thu, 29 May 2025 07:27:33 GMT
• Title: Promptus: Can Prompts Streaming Replace Video Streaming with Stable Diffusion
• Abstract summary: We propose Promptus, a disruptive communication system that streaming prompts instead of video content.<n>It represents real-world video frames with a series of "prompts" for delivery and employs Stable Diffusion to generate videos at the receiver.<n>Our work opens up a new paradigm for efficient video communication.
• Score: 6.093768205619601
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the exponential growth of video traffic, traditional video streaming systems are approaching their limits in compression efficiency and communication capacity. To further reduce bitrate while maintaining quality, we propose Promptus, a disruptive semantic communication system that streaming prompts instead of video content, which represents real-world video frames with a series of "prompts" for delivery and employs Stable Diffusion to generate videos at the receiver. To ensure that the generated video is pixel-aligned with the original video, a gradient descent-based prompt fitting framework is proposed. Further, a low-rank decomposition-based bitrate control algorithm is introduced to achieve adaptive bitrate. For inter-frame compression, an interpolation-aware fitting algorithm is proposed. Evaluations across various video genres demonstrate that, compared to H.265, Promptus can achieve more than a 4x bandwidth reduction while preserving the same perceptual quality. On the other hand, at extremely low bitrates, Promptus can enhance the perceptual quality by 0.139 and 0.118 (in LPIPS) compared to VAE and H.265, respectively, and decreases the ratio of severely distorted frames by 89.3% and 91.7%. Our work opens up a new paradigm for efficient video communication. Promptus is open-sourced at: https://github.com/JiangkaiWu/Promptus.

Related papers

• ScalablePromptus: Scalable and High-Fidelity Prompt-Based Video Streaming [51.48008871921326]
  Prompt-based video streaming transmits compact semantic prompts instead of pixel-level content for generative reconstruction.<n>We propose ScalablePromptus, which enhances Promptus with semantic and color-aware prompt inversion, spherical linear for intermediate frames, and--most critically--a dropout training strategy that produces rank-ordered prompt representations.
  arXiv  Detail & Related papers  (2026-07-28T08:59:39Z)
• Streaming of rendered content with adaptive frame rate and resolution [6.34507331016388]
  Existing solutions render content on a server at a fixed frame rate, typically 30 or 60 frames per second, and reduce resolution when bandwidth is restricted.<n>We exploit limits of human visual system to improve perceived quality while reducing rendering costs by adaptively adjusting both frame rate and resolution based on scene content and motion.<n>We propose a system in which a lightweight neural network predicts the optimal combination of frame rate and resolution for a given transmission bandwidth, content and motion velocity.
  arXiv  Detail & Related papers  (2026-05-09T14:48:10Z)
• Generative Video Compression: Towards 0.01% Compression Rate for Video Transmission [43.034936334574155]
  We introduce Generative Video Compression (GVC), a new framework that redefines the limits of video compression.<n>GVC encodes video into compact representations and delegates content reconstruction to the receiver.<n>Within the AI Flow framework, GVC opens new possibility for video communication in bandwidth- and resource-constrained environments.
  arXiv  Detail & Related papers  (2025-12-30T15:41:33Z)
• Plug-and-Play Versatile Compressed Video Enhancement [57.62582951699999]
  Video compression effectively reduces the size of files, making it possible for real-time cloud computing. However, it comes at the cost of visual quality, challenges the robustness of downstream vision models. We present a versatile-aware enhancement framework that adaptively enhance videos under different compression settings.
  arXiv  Detail & Related papers  (2025-04-21T18:39:31Z)
• Embedding Compression Distortion in Video Coding for Machines [67.97469042910855]
  Currently, video transmission serves not only the Human Visual System (HVS) for viewing but also machine perception for analysis.<n>We propose a Compression Distortion Embedding (CDRE) framework, which extracts machine-perception-related distortion representation and embeds it into downstream models.<n>Our framework can effectively boost the rate-task performance of existing codecs with minimal overhead in terms of execution time, and number of parameters.
  arXiv  Detail & Related papers  (2025-03-27T13:01:53Z)
• PromptMobile: Efficient Promptus for Low Bandwidth Mobile Video Streaming [18.32444457609669]
  This paper presents PromptMobile, an efficient acceleration framework tailored for on-device Promptus. We propose a two-stage efficient generation framework to reduce computational cost by 8.1x, (2) a fine-grained inter-frame caching to reduce redundant computations by 16.6%, (3) system-level optimizations to further enhance efficiency. Compared with other streaming methods, PromptMobile achives an average LPIPS improvement of 0.016 (compared with H.265), reducing 60% of severely distorted frames (compared to VQGAN)
  arXiv  Detail & Related papers  (2025-03-20T13:00:36Z)
• Enhancing Multi-Text Long Video Generation Consistency without Tuning: Time-Frequency Analysis, Prompt Alignment, and Theory [92.1714656167712]
  We propose a temporal Attention Reweighting Algorithm (TiARA) to enhance the consistency and coherence of videos generated with either single or multiple prompts. Our method is supported by a theoretical guarantee, the first-of-its-kind for frequency-based methods in diffusion models. For videos generated by multiple prompts, we further investigate key factors affecting prompt quality and propose PromptBlend, an advanced video prompt pipeline.
  arXiv  Detail & Related papers  (2024-12-23T03:56:27Z)
• Adaptive Caching for Faster Video Generation with Diffusion Transformers [52.73348147077075]
  Diffusion Transformers (DiTs) rely on larger models and heavier attention mechanisms, resulting in slower inference speeds. We introduce a training-free method to accelerate video DiTs, termed Adaptive Caching (AdaCache) We also introduce a Motion Regularization (MoReg) scheme to utilize video information within AdaCache, controlling the compute allocation based on motion content.
  arXiv  Detail & Related papers  (2024-11-04T18:59:44Z)
• Implicit Neural Representation for Videos Based on Residual Connection [0.0]
  We propose a method that uses low-resolution frames as residual connection that is considered effective for image reconstruction. Experimental results show that our method outperforms the existing method, HNeRV, in PSNR for 46 of the 49 videos.
  arXiv  Detail & Related papers  (2024-06-15T10:10:48Z)
• Leveraging Compressed Frame Sizes For Ultra-Fast Video Classification [12.322783570127756]
  Classifying videos into distinct categories, such as Sport and Music Video, is crucial for multimedia understanding and retrieval. Traditional methods require video decompression to extract pixel-level features like color, texture, and motion. We present a novel approach that examines only the post-compression bitstream of a video to perform classification, eliminating the need for bitstream decoding.
  arXiv  Detail & Related papers  (2024-03-13T14:35:13Z)
• Accelerated Event-Based Feature Detection and Compression for Surveillance Video Systems [1.5390526524075634]
  We propose a novel system which conveys temporal redundancy within a sparse decompressed representation. We leverage a video representation framework called ADDER to transcode framed videos to sparse, asynchronous intensity samples. Our work paves the way for upcoming neuromorphic sensors and is amenable to future applications with spiking neural networks.
  arXiv  Detail & Related papers  (2023-12-13T15:30:29Z)
• Perceptual Quality Improvement in Videoconferencing using Keyframes-based GAN [28.773037051085318]
  We propose a novel GAN-based method for compression artifacts reduction in videoconferencing. First, we extract multi-scale features from the compressed and reference frames. Then, our architecture combines these features in a progressive manner according to facial landmarks.
  arXiv  Detail & Related papers  (2023-11-07T16:38:23Z)
• ReBotNet: Fast Real-time Video Enhancement [59.08038313427057]
  Most restoration networks are slow, have high computational bottleneck, and can't be used for real-time video enhancement. In this work, we design an efficient and fast framework to perform real-time enhancement for practical use-cases like live video calls and video streams. To evaluate our method, we emulate two new datasets that real-world video call and streaming scenarios, and show extensive results on multiple datasets where ReBotNet outperforms existing approaches with lower computations, reduced memory requirements, and faster inference time.
  arXiv  Detail & Related papers  (2023-03-23T17:58:05Z)
• Compressed Vision for Efficient Video Understanding [83.97689018324732]
  We propose a framework enabling research on hour-long videos with the same hardware that can now process second-long videos. We replace standard video compression, e.g. JPEG, with neural compression and show that we can directly feed compressed videos as inputs to regular video networks.
  arXiv  Detail & Related papers  (2022-10-06T15:35:49Z)
• Leveraging Bitstream Metadata for Fast, Accurate, Generalized Compressed Video Quality Enhancement [74.1052624663082]
  We develop a deep learning architecture capable of restoring detail to compressed videos. We show that this improves restoration accuracy compared to prior compression correction methods. We condition our model on quantization data which is readily available in the bitstream.
  arXiv  Detail & Related papers  (2022-01-31T18:56:04Z)
• FREGAN : an application of generative adversarial networks in enhancing the frame rate of videos [1.1688030627514534]
  FREGAN (Frame Rate Enhancement Generative Adversarial Network) model has been proposed, which predicts future frames of a video sequence based on a sequence of past frames. We have validated the effectiveness of the proposed model on the standard datasets. The experimental outcomes illustrate that the proposed model has a Peak signal-to-noise ratio (PSNR) of 34.94 and a Structural Similarity Index (SSIM) of 0.95.
  arXiv  Detail & Related papers  (2021-11-01T17:19:00Z)
• Efficient Video Compression via Content-Adaptive Super-Resolution [11.6624528293976]
  Video compression is a critical component of Internet video delivery. Recent work has shown that deep learning techniques can rival or outperform human algorithms. This paper presents a new approach that augments a recent deep learning-based video compression scheme.
  arXiv  Detail & Related papers  (2021-04-06T07:01:06Z)
• Prediction-assistant Frame Super-Resolution for Video Streaming [40.60863957681011]
  We propose to enhance video quality using lossy frames in two situations. For the first case, we propose a small yet effective video frame prediction network. For the second case, we improve the video prediction network to associate current frames as well as previous frames to restore high-quality images.
  arXiv  Detail & Related papers  (2021-03-17T06:05:27Z)
• Ultra-low bitrate video conferencing using deep image animation [7.263312285502382]
  We propose a novel deep learning approach for ultra-low video compression for video conferencing applications. We employ deep neural networks to encode motion information as keypoint displacement and reconstruct the video signal at the decoder side.
  arXiv  Detail & Related papers  (2020-12-01T09:06:34Z)
• Conditional Entropy Coding for Efficient Video Compression [82.35389813794372]
  We propose a very simple and efficient video compression framework that only focuses on modeling the conditional entropy between frames. We first show that a simple architecture modeling the entropy between the image latent codes is as competitive as other neural video compression works and video codecs. We then propose a novel internal learning extension on top of this architecture that brings an additional 10% savings without trading off decoding speed.
  arXiv  Detail & Related papers  (2020-08-20T20:01:59Z)
• Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model [164.7489982837475]
  This paper proposes a Recurrent Learned Video Compression (RLVC) approach with the Recurrent Auto-Encoder (RAE) and Recurrent Probability Model ( RPM) The RAE employs recurrent cells in both the encoder and decoder to exploit the temporal correlation among video frames. Our approach achieves the state-of-the-art learned video compression performance in terms of both PSNR and MS-SSIM.
  arXiv  Detail & Related papers  (2020-06-24T08:46:33Z)
• Content Adaptive and Error Propagation Aware Deep Video Compression [110.31693187153084]
  We propose a content adaptive and error propagation aware video compression system. Our method employs a joint training strategy by considering the compression performance of multiple consecutive frames instead of a single frame. Instead of using the hand-crafted coding modes in the traditional compression systems, we design an online encoder updating scheme in our system.
  arXiv  Detail & Related papers  (2020-03-25T09:04:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.