Beyond Wide-Angle Images: Structure-to-Detail Video Portrait Correction via Unsupervised Spatiotemporal Adaptation

• URL: http://arxiv.org/abs/2504.00401v2
• Date: Wed, 06 Aug 2025 04:43:38 GMT
• Title: Beyond Wide-Angle Images: Structure-to-Detail Video Portrait Correction via Unsupervised Spatiotemporal Adaptation
• Authors: Wenbo Nie, Lang Nie, Chunyu Lin, Jingwen Chen, Ke Xing, Jiyuan Wang, Kang Liao,
• Abstract summary: We propose a structure-to-detail portrait correction model named ImagePC.<n>It integrates the long-range awareness of the transformer and multi-step denoising of diffusion models into a unified framework.<n>Considering the high cost of obtaining video labels, we then repurpose ImagePC for unlabeled wide-angle videos (termed VideoPC)
• Score: 30.50527916658749
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Wide-angle cameras, despite their popularity for content creation, suffer from distortion-induced facial stretching-especially at the edge of the lens-which degrades visual appeal. To address this issue, we propose a structure-to-detail portrait correction model named ImagePC. It integrates the long-range awareness of the transformer and multi-step denoising of diffusion models into a unified framework, achieving global structural robustness and local detail refinement. Besides, considering the high cost of obtaining video labels, we then repurpose ImagePC for unlabeled wide-angle videos (termed VideoPC), by spatiotemporal diffusion adaption with spatial consistency and temporal smoothness constraints. For the former, we encourage the denoised image to approximate pseudo labels following the wide-angle distortion distribution pattern, while for the latter, we derive rectification trajectories with backward optical flows and smooth them. Compared with ImagePC, VideoPC maintains high-quality facial corrections in space and mitigates the potential temporal shakes sequentially in blind scenarios. Finally, to establish an evaluation benchmark and train the framework, we establish a video portrait dataset with a large diversity in the number of people, lighting conditions, and background. Experiments demonstrate that the proposed methods outperform existing solutions quantitatively and qualitatively, contributing to high-fidelity wide-angle videos with stable and natural portraits. The codes and dataset will be available.

Related papers

• GaVS: 3D-Grounded Video Stabilization via Temporally-Consistent Local Reconstruction and Rendering [54.489285024494855]
  Video stabilization is pivotal for video processing, as it removes unwanted shakiness while preserving the original user motion intent.<n>Existing approaches, depending on the domain they operate, suffer from several issues that degrade the user experience.<n>We introduce textbfGaVS, a novel 3D-grounded approach that reformulates video stabilization as a temporally-consistent local reconstruction and rendering' paradigm.
  arXiv  Detail & Related papers  (2025-06-30T15:24:27Z)
• DicFace: Dirichlet-Constrained Variational Codebook Learning for Temporally Coherent Video Face Restoration [24.004683996460685]
  Video face restoration faces a critical challenge in maintaining temporal consistency while recovering facial details from degraded inputs.<n>This paper presents a novel approach that extends Vector-Quantized Variational Autoencoders (VQ-VAEs), pretrained on static high-quality images, into a video restoration framework.
  arXiv  Detail & Related papers  (2025-06-16T10:54:28Z)
• VipDiff: Towards Coherent and Diverse Video Inpainting via Training-free Denoising Diffusion Models [21.584843961386888]
  VipDiff is a framework for conditioning diffusion model on the reverse diffusion process to produce temporal-coherent inpainting results.<n>It can largely outperform state-of-the-art video inpainting methods in terms of both spatial-temporal coherence and fidelity.
  arXiv  Detail & Related papers  (2025-01-21T16:39:09Z)
• DiffuEraser: A Diffusion Model for Video Inpainting [13.292164408616257]
  We introduce DiffuEraser, a video inpainting model based on stable diffusion, to fill masked regions with greater details and more coherent structures.<n>We also expand the temporal receptive fields of both the prior model and DiffuEraser, and further enhance consistency by leveraging the temporal smoothing property of Video Diffusion Models.
  arXiv  Detail & Related papers  (2025-01-17T08:03:02Z)
• VISION-XL: High Definition Video Inverse Problem Solver using Latent Image Diffusion Models [58.464465016269614]
  We propose a novel framework for solving high-definition video inverse problems using latent image diffusion models.<n>Our approach delivers HD-resolution reconstructions in under 6 seconds per frame on a single NVIDIA 4090 GPU.
  arXiv  Detail & Related papers  (2024-11-29T08:10:49Z)
• Optical-Flow Guided Prompt Optimization for Coherent Video Generation [51.430833518070145]
  We propose a framework called MotionPrompt that guides the video generation process via optical flow.<n>We optimize learnable token embeddings during reverse sampling steps by using gradients from a trained discriminator applied to random frame pairs.<n>This approach allows our method to generate visually coherent video sequences that closely reflect natural motion dynamics, without compromising the fidelity of the generated content.
  arXiv  Detail & Related papers  (2024-11-23T12:26:52Z)
• Pixel-Aligned Multi-View Generation with Depth Guided Decoder [86.1813201212539]
  We propose a novel method for pixel-level image-to-multi-view generation. Unlike prior work, we incorporate attention layers across multi-view images in the VAE decoder of a latent video diffusion model. Our model enables better pixel alignment across multi-view images.
  arXiv  Detail & Related papers  (2024-08-26T04:56:41Z)
• MultiDiff: Consistent Novel View Synthesis from a Single Image [60.04215655745264]
  MultiDiff is a novel approach for consistent novel view synthesis of scenes from a single RGB image. Our results demonstrate that MultiDiff outperforms state-of-the-art methods on the challenging, real-world datasets RealEstate10K and ScanNet.
  arXiv  Detail & Related papers  (2024-06-26T17:53:51Z)
• FlexNeRF: Photorealistic Free-viewpoint Rendering of Moving Humans from Sparse Views [71.77680030806513]
  We present FlexNeRF, a method for photorealistic freeviewpoint rendering of humans in motion from monocular videos. Our approach works well with sparse views, which is a challenging scenario when the subject is exhibiting fast/complex motions. Thanks to our novel temporal and cyclic consistency constraints, our approach provides high quality outputs as the observed views become sparser.
  arXiv  Detail & Related papers  (2023-03-25T05:47:08Z)
• Video Demoireing with Relation-Based Temporal Consistency [68.20281109859998]
  Moire patterns, appearing as color distortions, severely degrade image and video qualities when filming a screen with digital cameras. We study how to remove such undesirable moire patterns in videos, namely video demoireing.
  arXiv  Detail & Related papers  (2022-04-06T17:45:38Z)
• Correcting Face Distortion in Wide-Angle Videos [85.88898349347149]
  We present a video warping algorithm to correct these distortions. Our key idea is to apply stereographic projection locally on the facial regions. For performance evaluation, we develop a wide-angle video dataset with a wide range of focal lengths.
  arXiv  Detail & Related papers  (2021-11-18T21:28:17Z)
• Practical Wide-Angle Portraits Correction with Deep Structured Models [17.62752136436382]
  This paper introduces the first deep learning based approach to remove perspective distortions from photos. Given a wide-angle portrait as input, we build a cascaded network consisting of a LineNet, a ShapeNet, and a transition module. For the quantitative evaluation, we introduce two novel metrics, line consistency and face congruence.
  arXiv  Detail & Related papers  (2021-04-26T10:47:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.