Understanding Deformable Alignment in Video Super-Resolution

• URL: http://arxiv.org/abs/2009.07265v1
• Date: Tue, 15 Sep 2020 17:55:06 GMT
• Title: Understanding Deformable Alignment in Video Super-Resolution
• Abstract summary: We show that deformable convolution can be decomposed into a combination of spatial warping and convolution. We propose an offset-fidelity loss that guides the offset learning with optical flow.
• Score: 90.9481291865295
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deformable convolution, originally proposed for the adaptation to geometric variations of objects, has recently shown compelling performance in aligning multiple frames and is increasingly adopted for video super-resolution. Despite its remarkable performance, its underlying mechanism for alignment remains unclear. In this study, we carefully investigate the relation between deformable alignment and the classic flow-based alignment. We show that deformable convolution can be decomposed into a combination of spatial warping and convolution. This decomposition reveals the commonality of deformable alignment and flow-based alignment in formulation, but with a key difference in their offset diversity. We further demonstrate through experiments that the increased diversity in deformable alignment yields better-aligned features, and hence significantly improves the quality of video super-resolution output. Based on our observations, we propose an offset-fidelity loss that guides the offset learning with optical flow. Experiments show that our loss successfully avoids the overflow of offsets and alleviates the instability problem of deformable alignment. Aside from the contributions to deformable alignment, our formulation inspires a more flexible approach to introduce offset diversity to flow-based alignment, improving its performance.

Related papers

• Neural Video Compression Based on Deformable Temporal Alignment and Difference-aware Fusion [1.8740972654414068]
  Method combines deformable temporal alignment and difference-aware spatial selective fusion.<n>Experiments show that the proposed method achieves certain rate-distortion performance improve- ment over DCVC-DC.
  arXiv  Detail & Related papers  (2026-09-03T08:20:19Z)
• SPDCN: Strip-based Deformable Convolutional Network for Steel Surface Defect Segmentation [8.936780165911612]
  Strip-based Predictor for Deformable Convolutional Networks (SPDCN)<n>textbfFuzzy-enhanced Multi-scale Context Module (FMCM)<n>textbf Direction-Aware Deformable Convolution (ADADC)
  arXiv  Detail & Related papers  (2026-07-23T15:59:10Z)
• Hierarchical Consistency Learning for Test-time Adaptation in Camouflage Perception [50.278200968044665]
  Camouflaged object detection (COD) aims to localize targets that exhibit minimal perceptual differences from backgrounds through physical attributes.<n>Existing methods, constrained by the static train-then-freeze paradigm, suffer from domain rigidity and dependency annotation.<n>We propose the hierarchical consistency learning framework, which integrates test-time adaptation for dynamic representation recalibration.
  arXiv  Detail & Related papers  (2026-05-25T09:57:46Z)
• Probability-Conserving Flow Guidance [49.03107678763765]
  Adaptive Manifold Guidance (AdaMaG) improves realism, reduces hallucinations, and controlled desaturation in high-guidance.<n>Across image generation benchmarks, AdaMaG improves realism, reduces hallucinations, and controlled desaturation in high-guidance.
  arXiv  Detail & Related papers  (2026-05-19T16:34:01Z)
• Content Adaptive based Motion Alignment Framework for Learned Video Compression [72.13599533975413]
  This paper proposes a content adaptive based motion alignment framework.<n>We first introduce a two-stage flow-guided deformable warping mechanism that refines motion compensation with coarse-to-fine offset prediction and mask modulation.<n>Second, we propose a multi-reference quality aware strategy that adjusts distortion weights based on reference quality, and applies it to hierarchical training to reduce error propagation.<n>Third, we integrate a training-free module that downsamples frames by motion magnitude and resolution to obtain smooth motion estimation.
  arXiv  Detail & Related papers  (2025-12-15T02:51:47Z)
• Robust Posterior Diffusion-based Sampling via Adaptive Guidance Scale [39.27744518020771]
  We propose an adaptive likelihood step-size strategy to guide the diffusion process for inverse-problem formulations.<n>The resulting approach, Adaptive Posterior diffusion Sampling (AdaPS), is hyper-free and improves reconstruction quality across diverse imaging tasks.
  arXiv  Detail & Related papers  (2025-11-23T14:37:59Z)
• OSCAR: Orthogonal Stochastic Control for Alignment-Respecting Diversity in Flow Matching [14.664226708184676]
  Flow-based text-to-image models follow deterministic trajectories, forcing users to repeatedly sample to discover diverse modes.<n>We present a training-free, inference-time control mechanism that makes the flow itself diversity-aware.
  arXiv  Detail & Related papers  (2025-10-10T07:07:19Z)
• Solving Inverse Problems with FLAIR [59.02385492199431]
  Flow-based latent generative models are able to generate images with remarkable quality, even enabling text-to-image generation.<n>We present FLAIR, a novel training free variational framework that leverages flow-based generative models as a prior for inverse problems.<n>Results on standard imaging benchmarks demonstrate that FLAIR consistently outperforms existing diffusion- and flow-based methods in terms of reconstruction quality and sample diversity.
  arXiv  Detail & Related papers  (2025-06-03T09:29:47Z)
• Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences through $f$-divergence Minimization [9.242941447250969]
  This study focuses on extending reverse Kullback-Leibler divergence in the alignment paradigm of text-to-image models to $f$-divergence. We conduct comprehensive evaluation on image-text alignment performance, human value alignment performance and generation diversity performance under different divergence constraints.
  arXiv  Detail & Related papers  (2024-09-15T15:46:03Z)
• Controlled Learning of Pointwise Nonlinearities in Neural-Network-Like Architectures [14.93489065234423]
  We present a general variational framework for the training of freeform nonlinearities in layered computational architectures. The slope constraints allow us to impose properties such as 1-Lipschitz stability, firm non-expansiveness, and monotonicity/invertibility. We show how to solve the numerically function-optimization problem by representing the nonlinearities in a suitable (nonuniform) B-spline basis.
  arXiv  Detail & Related papers  (2024-08-23T14:39:27Z)
• Collaborative Feedback Discriminative Propagation for Video Super-Resolution [66.61201445650323]
  Key success of video super-resolution (VSR) methods stems mainly from exploring spatial and temporal information. Inaccurate alignment usually leads to aligned features with significant artifacts. propagation modules only propagate the same timestep features forward or backward.
  arXiv  Detail & Related papers  (2024-04-06T22:08:20Z)
• Semi-Supervised Coupled Thin-Plate Spline Model for Rotation Correction and Beyond [84.56978780892783]
  We propose CoupledTPS, which iteratively couples multiple TPS with limited control points into a more flexible and powerful transformation. In light of the laborious annotation cost, we develop a semi-supervised learning scheme to improve warping quality by exploiting unlabeled data. Experiments demonstrate the superiority and universality of CoupledTPS over the existing state-of-the-art solutions for rotation correction.
  arXiv  Detail & Related papers  (2024-01-24T13:03:28Z)
• Training-Free Semantic Video Composition via Pre-trained Diffusion Model [96.0168609879295]
  Current approaches, predominantly trained on videos with adjusted foreground color and lighting, struggle to address deep semantic disparities beyond superficial adjustments. We propose a training-free pipeline employing a pre-trained diffusion model imbued with semantic prior knowledge. Experimental results reveal that our pipeline successfully ensures the visual harmony and inter-frame coherence of the outputs.
  arXiv  Detail & Related papers  (2024-01-17T13:07:22Z)
• Linearization Algorithms for Fully Composite Optimization [61.20539085730636]
  This paper studies first-order algorithms for solving fully composite optimization problems convex compact sets. We leverage the structure of the objective by handling differentiable and non-differentiable separately, linearizing only the smooth parts.
  arXiv  Detail & Related papers  (2023-02-24T18:41:48Z)
• Learned Video Compression via Heterogeneous Deformable Compensation Network [78.72508633457392]
  We propose a learned video compression framework via heterogeneous deformable compensation strategy (HDCVC) to tackle the problems of unstable compression performance. More specifically, the proposed algorithm extracts features from the two adjacent frames to estimate content-Neighborhood heterogeneous deformable (HetDeform) kernel offsets. Experimental results indicate that HDCVC achieves superior performance than the recent state-of-the-art learned video compression approaches.
  arXiv  Detail & Related papers  (2022-07-11T02:31:31Z)
• FDAN: Flow-guided Deformable Alignment Network for Video Super-Resolution [12.844337773258678]
  Flow-guided Deformable Module (FDM) is proposed to integrate optical flow into deformable convolution. FDAN reaches the state-of-the-art performance on two benchmark datasets.
  arXiv  Detail & Related papers  (2021-05-12T13:18:36Z)
• Learning Disentangled Feature Representation for Hybrid-distorted Image Restoration [41.99534893489878]
  We introduce the concept of Disentangled Feature Learning to achieve the feature-level divide-and-conquer of hybrid distortions. Specifically, we propose the feature disentanglement module (FDM) to distribute feature representations of different distortions into different channels. We also propose a feature aggregation module (FAM) with channel-wise attention to adaptively filter out the distortion representations.
  arXiv  Detail & Related papers  (2020-07-22T13:43:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.