LiteVSR: Lightweight Adaptation of Frozen Diffusion Transformers for Video Super-Resolution

• URL: http://arxiv.org/abs/2606.09250v1
• Date: Mon, 08 Jun 2026 09:23:42 GMT
• Title: LiteVSR: Lightweight Adaptation of Frozen Diffusion Transformers for Video Super-Resolution
• Abstract summary: We propose a minimalist framework that performs Video Super-Resolution (VSR) using a frozen Transformer with a lightweight State-Aware Adapter.<n> LiteVSR achieves competitive restoration quality with only 11.25% trainable parameters and 12 GPU-hours of training on a single A100, while maintaining fast sampling (down to a single step) compatibility.
• Score: 73.85089704122315
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adapting large-scale pre-trained video generators for Video Super-Resolution (VSR) in novel domains remains computationally prohibitive. Methods that reformulate generation as direct Low-Quality to High-Quality mappings deviate from the original generative formulation, demanding extensive fine-tuning. ControlNet-style adapters lose their efficiency under modern Diffusion Transformers since the absence of encoder-decoder hierarchy forces duplication of the entire backbone. We observe that flow matching offers a principled alternative for cross-domain VSR adaptation. By predicting a constant velocity field across all timesteps, the adaptation task reduces to learning a fixed injection pattern rather than time-varying transformations. Building on this insight, we propose LiteVSR, a minimalist framework that performs VSR using a completely frozen Diffusion Transformer with a lightweight State-Aware Adapter. The adapter employs a dual-stream architecture that extracts static structural cues from the LQ input and dynamic cues from intermediate denoising states, aligning them through time-dependent cross-attention to enable adaptive transition from structural alignment to texture refinement as denoising proceeds. LiteVSR achieves competitive restoration quality with only 11.25% trainable parameters and 12 GPU-hours of training on a single A100, while maintaining fast sampling (down to a single step) compatibility.

Related papers

• FreeTransformSR: Efficient Lightweight Image Super-Resolution via Free Low-Rank Learnable Transform [0.549402648081308]
  Single image super-resolution aims to reconstruct high-resolution images from low-resolution inputs.<n>This paper proposes FreeTransformSR, a novel lightweight super-resolution network based on a channel-wise free low-rank learnable transform.<n>Experiments on five benchmark datasets demonstrate that FreeTransformSR achieves competitive PSNR/SSIM performance with significantly fewer parameters and FLOPs.
  arXiv  Detail & Related papers  (2026-09-05T06:12:53Z)
• TRaM-VSR: Importance-Aware Token Routing and Merging for One-Step Diffusion Video Super-Resolution [52.96050253456634]
  TRaM-VSR is a framework for adaptive token allocation using context-aware video priors and network-levels.<n>It is shown that TRaM-VSR accelerates inference significantly while preserving state-of-the-art reconstruction quality and robust temporal consistency.
  arXiv  Detail & Related papers  (2026-07-24T11:57:26Z)
• Beyond Quadratic: Linear-Time Change Detection with RWKV [48.3313296326424]
  ChangeRWKV is a new architecture for remote sensing change detection.<n>It combines the parallelizable training of Transformers with the linear-time inference of RNNs.<n>It achieves state-of-the-art performance on the LEVIR-CD benchmark, with an 85.46% IoU and 92.16% F1 score.
  arXiv  Detail & Related papers  (2026-03-20T03:23:02Z)
• VAE-REPA: Variational Autoencoder Representation Alignment for Efficient Diffusion Training [53.09658039757408]
  This paper proposes textbfnamex, a lightweight intrinsic guidance framework for efficient diffusion training.<n>name aligns the intermediate latent features of diffusion transformers with VAE features via a lightweight projection layer, supervised by a feature alignment loss.<n>Experiments demonstrate that name improves both generation quality and training convergence speed compared to vanilla diffusion transformers.
  arXiv  Detail & Related papers  (2026-01-25T13:22:38Z)
• FlashVSR: Towards Real-Time Diffusion-Based Streaming Video Super-Resolution [61.284842030283464]
  FlashVSR is the first diffusion-based one-step streaming framework towards real-time VSR.<n>It runs at approximately 17 FPS for 768x1408 videos on a single A100 GPU.<n>It scales reliably to ultra-high resolutions and achieves state-of-the-art performance with up to 12x speedup over prior one-step diffusion VSR models.
  arXiv  Detail & Related papers  (2025-10-14T17:25:54Z)
• InfVSR: Breaking Length Limits of Generic Video Super-Resolution [40.30527504651693]
  InfVSR is an autoregressive-one-step-diffusion paradigm for long sequences.<n>We distill the diffusion process into a single step efficiently, with patch-wise pixel supervision and cross-chunk distribution matching.<n>Our method pushes the frontier of long-form VSR, achieves state-of-the-art quality with enhanced semantic consistency, and delivers up to 58x speed-up over existing methods.
  arXiv  Detail & Related papers  (2025-10-01T14:21:45Z)
• RAPID^3: Tri-Level Reinforced Acceleration Policies for Diffusion Transformer [86.57077884971478]
  Diffusion Transformers (DiTs) excel at visual generation yet remain hampered by slow sampling.<n>We introduce RAPID3: Tri-Level Reinforced Acceleration Policies for Diffusion Transformers.<n>It delivers image-wise acceleration with zero updates to the base generator.<n>It achieves nearly 3x faster sampling with competitive generation quality.
  arXiv  Detail & Related papers  (2025-09-26T13:20:52Z)
• AdaRing: Towards Ultra-Light Vision-Language Adaptation via Cross-Layer Tensor Ring Decomposition [41.654675205772485]
  We propose a vision-language fine-tuning framework based on cross-layer tensor ring decomposition (TRD) with the integration and collaboration of diverse adapters, called AdaRing.<n>Our experiments show that the proposed AdaRing achieves the state-of-the-art performance while reducing average training parameters by 90%.
  arXiv  Detail & Related papers  (2025-08-16T01:56:27Z)
• Video Super-Resolution Transformer [85.11270760456826]
  Video super-resolution (VSR), with the aim to restore a high-resolution video from its corresponding low-resolution version, is a spatial-temporal sequence prediction problem. Recently, Transformer has been gaining popularity due to its parallel computing ability for sequence-to-sequence modeling. In this paper, we present a spatial-temporal convolutional self-attention layer with a theoretical understanding to exploit the locality information.
  arXiv  Detail & Related papers  (2021-06-12T20:00:32Z)
• Conv-Transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-End Speech Recognition [8.046120977786702]
  Transformer has achieved competitive performance against state-of-the-art end-to-end models in automatic speech recognition (ASR) The original Transformer, with encoder-decoder architecture, is only suitable for offline ASR. We show that this architecture, named Conv-Transformer Transducer, achieves competitive performance on LibriSpeech dataset (3.6% WER on test-clean) without external language models.
  arXiv  Detail & Related papers  (2020-08-13T08:20:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.