ChopGrad: Pixel-Wise Losses for Latent Video Diffusion via Truncated Backpropagation

• URL: http://arxiv.org/abs/2603.17812v1
• Date: Wed, 18 Mar 2026 15:04:57 GMT
• Title: ChopGrad: Pixel-Wise Losses for Latent Video Diffusion via Truncated Backpropagation
• Abstract summary: This paper introduces ChopGrad, a truncated backpropagation scheme for video decoding, limiting gradient to local frame windows while maintaining global consistency.<n>We provide a theoretical analysis of this approximation and show that it enables efficient fine-tuning with frame-wise losses.
• Score: 33.672422379245766
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent video diffusion models achieve high-quality generation through recurrent frame processing where each frame generation depends on previous frames. However, this recurrent mechanism means that training such models in the pixel domain incurs prohibitive memory costs, as activations accumulate across the entire video sequence. This fundamental limitation also makes fine-tuning these models with pixel-wise losses computationally intractable for long or high-resolution videos. This paper introduces ChopGrad, a truncated backpropagation scheme for video decoding, limiting gradient computation to local frame windows while maintaining global consistency. We provide a theoretical analysis of this approximation and show that it enables efficient fine-tuning with frame-wise losses. ChopGrad reduces training memory from scaling linearly with the number of video frames (full backpropagation) to constant memory, and compares favorably to existing state-of-the-art video diffusion models across a suite of conditional video generation tasks with pixel-wise losses, including video super-resolution, video inpainting, video enhancement of neural-rendered scenes, and controlled driving video generation.

Related papers

• Latent Inter-Frame Pruning: A Training-Free Method Bridging Traditional Video Compression and Modern Diffusion Transformers for Efficient Generation [2.639628765667673]
  Video generation, while capable of generating realistic videos, is computationally expensive and slow, prohibiting real-time applications.<n>We observe that video latents encoded via an autoencoder contain redundancy along the temporal axis.<n>We propose the Latent Inter-frame Pruning framework to prune (skip the re-computation of) duplicated latent patches, thereby reducing computational burden and increasing throughput.
  arXiv  Detail & Related papers  (2026-04-26T20:03:11Z)
• Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models [63.99949971803903]
  We present a neural network structure, FramePack, to train next-frame (or next-frame-section) prediction models for video generation.<n>FramePack compresses input frame contexts with frame-wise importance so that more frames can be encoded within a fixed context length.<n>We show that existing video diffusion models can be finetuned with FramePack, and analyze the differences between different packing schedules.
  arXiv  Detail & Related papers  (2025-04-17T04:02:31Z)
• VideoLifter: Lifting Videos to 3D with Fast Hierarchical Stereo Alignment [54.66217340264935]
  VideoLifter is a novel video-to-3D pipeline that leverages a local-to-global strategy on a fragment basis.<n>It significantly accelerates the reconstruction process, reducing training time by over 82% while holding better visual quality than current SOTA methods.
  arXiv  Detail & Related papers  (2025-01-03T18:52:36Z)
• High-Efficiency Neural Video Compression via Hierarchical Predictive Learning [27.41398149573729]
  Enhanced Deep Hierarchical Video Compression-DHVC 2.0- introduces superior compression performance and impressive complexity efficiency. Uses hierarchical predictive coding to transform each video frame into multiscale representations. Supports transmission-friendly progressive decoding, making it particularly advantageous for networked video applications in the presence of packet loss.
  arXiv  Detail & Related papers  (2024-10-03T15:40:58Z)
• Predictive Coding For Animation-Based Video Compression [13.161311799049978]
  We propose a predictive coding scheme which uses image animation as a predictor, and codes the residual with respect to the actual target frame. Our experiments indicate a significant gain, in excess of 70% compared to the HEVC video standard and over 30% compared to VVC.
  arXiv  Detail & Related papers  (2023-07-09T14:40:54Z)
• Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation [93.18163456287164]
  This paper proposes a novel text-guided video-to-video translation framework to adapt image models to videos. Our framework achieves global style and local texture temporal consistency at a low cost.
  arXiv  Detail & Related papers  (2023-06-13T17:52:23Z)
• Recurrent Video Restoration Transformer with Guided Deformable Attention [116.1684355529431]
  We propose RVRT, which processes local neighboring frames in parallel within a globally recurrent framework. RVRT achieves state-of-the-art performance on benchmark datasets with balanced model size, testing memory and runtime.
  arXiv  Detail & Related papers  (2022-06-05T10:36:09Z)
• Learning Trajectory-Aware Transformer for Video Super-Resolution [50.49396123016185]
  Video super-resolution aims to restore a sequence of high-resolution (HR) frames from their low-resolution (LR) counterparts. Existing approaches usually align and aggregate video frames from limited adjacent frames. We propose a novel Transformer for Video Super-Resolution (TTVSR)
  arXiv  Detail & Related papers  (2022-04-08T03:37:39Z)
• Efficient Video Segmentation Models with Per-frame Inference [117.97423110566963]
  We focus on improving the temporal consistency without introducing overhead in inference. We propose several techniques to learn from the video sequence, including a temporal consistency loss and online/offline knowledge distillation methods.
  arXiv  Detail & Related papers  (2022-02-24T23:51:36Z)
• VRT: A Video Restoration Transformer [126.79589717404863]
  Video restoration (e.g., video super-resolution) aims to restore high-quality frames from low-quality frames. We propose a Video Restoration Transformer (VRT) with parallel frame prediction and long-range temporal dependency modelling abilities.
  arXiv  Detail & Related papers  (2022-01-28T17:54:43Z)
• Memory-Augmented Non-Local Attention for Video Super-Resolution [61.55700315062226]
  We propose a novel video super-resolution method that aims at generating high-fidelity high-resolution (HR) videos from low-resolution (LR) ones. Previous methods predominantly leverage temporal neighbor frames to assist the super-resolution of the current frame. In contrast, we devise a cross-frame non-local attention mechanism that allows video super-resolution without frame alignment.
  arXiv  Detail & Related papers  (2021-08-25T05:12:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.