Panorama Generation From NFoV Image Done Right

• URL: http://arxiv.org/abs/2503.18420v1
• Date: Mon, 24 Mar 2025 08:04:33 GMT
• Title: Panorama Generation From NFoV Image Done Right
• Authors: Dian Zheng, Cheng Zhang, Xiao-Ming Wu, Cao Li, Chengfei Lv, Jian-Fang Hu, Wei-Shi Zheng,
• Abstract summary: We propose a distortion-specific CLIP, named Distort-CLIP, to evaluate the panorama distortion and discover the textbfvisual cheating'' phenomenon.<n>To address the phenomenon, we propose textbfPanoDecouple, a decoupled diffusion model framework, which decouples the panorama generation into distortion guidance and content completion.
• Score: 34.92299037497302
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generating 360-degree panoramas from narrow field of view (NFoV) image is a promising computer vision task for Virtual Reality (VR) applications. Existing methods mostly assess the generated panoramas with InceptionNet or CLIP based metrics, which tend to perceive the image quality and is \textbf{not suitable for evaluating the distortion}. In this work, we first propose a distortion-specific CLIP, named Distort-CLIP to accurately evaluate the panorama distortion and discover the \textbf{``visual cheating''} phenomenon in previous works (\ie, tending to improve the visual results by sacrificing distortion accuracy). This phenomenon arises because prior methods employ a single network to learn the distinct panorama distortion and content completion at once, which leads the model to prioritize optimizing the latter. To address the phenomenon, we propose \textbf{PanoDecouple}, a decoupled diffusion model framework, which decouples the panorama generation into distortion guidance and content completion, aiming to generate panoramas with both accurate distortion and visual appeal. Specifically, we design a DistortNet for distortion guidance by imposing panorama-specific distortion prior and a modified condition registration mechanism; and a ContentNet for content completion by imposing perspective image information. Additionally, a distortion correction loss function with Distort-CLIP is introduced to constrain the distortion explicitly. The extensive experiments validate that PanoDecouple surpasses existing methods both in distortion and visual metrics.

Related papers

• RoFIR: Robust Fisheye Image Rectification Framework Impervious to Optical Center Deviation [88.54817424560056]
  We propose a distortion vector map (DVM) that measures the degree and direction of local distortion. By learning the DVM, the model can independently identify local distortions at each pixel without relying on global distortion patterns. In the pre-training stage, it predicts the distortion vector map and perceives the local distortion features of each pixel. In the fine-tuning stage, it predicts a pixel-wise flow map for deviated fisheye image rectification.
  arXiv  Detail & Related papers  (2024-06-27T06:38:56Z)
• Möbius Transform for Mitigating Perspective Distortions in Representation Learning [43.86985901138407]
  Perspective distortion (PD) causes unprecedented changes in shape, size, orientation, angles, and other spatial relationships in images. We propose mitigating perspective distortion (MPD) by employing a fine-grained parameter control on a specific family of M"obius transform. We present a dedicated perspectively distorted benchmark dataset, ImageNet-PD, to benchmark the robustness of deep learning models against this new dataset.
  arXiv  Detail & Related papers  (2024-03-07T15:39:00Z)
• RecRecNet: Rectangling Rectified Wide-Angle Images by Thin-Plate Spline Model and DoF-based Curriculum Learning [62.86400614141706]
  We propose a new learning model, i.e., Rectangling Rectification Network (RecRecNet) Our model can flexibly warp the source structure to the target domain and achieves an end-to-end unsupervised deformation. Experiments show the superiority of our solution over the compared methods on both quantitative and qualitative evaluations.
  arXiv  Detail & Related papers  (2023-01-04T15:12:57Z)
• SIR: Self-supervised Image Rectification via Seeing the Same Scene from Multiple Different Lenses [82.56853587380168]
  We propose a novel self-supervised image rectification (SIR) method based on an important insight that the rectified results of distorted images of the same scene from different lens should be the same. We leverage a differentiable warping module to generate the rectified images and re-distorted images from the distortion parameters. Our method achieves comparable or even better performance than the supervised baseline method and representative state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-11-30T08:23:25Z)
• Generative and Discriminative Learning for Distorted Image Restoration [22.230017059874445]
  Liquify is a technique for image editing, which can be used for image distortion. We propose a novel generative and discriminative learning method based on deep neural networks.
  arXiv  Detail & Related papers  (2020-11-11T14:01:29Z)
• Wide-angle Image Rectification: A Survey [86.36118799330802]
  wide-angle images contain distortions that violate the assumptions underlying pinhole camera models. Image rectification, which aims to correct these distortions, can solve these problems. We present a detailed description and discussion of the camera models used in different approaches. Next, we review both traditional geometry-based image rectification methods and deep learning-based methods.
  arXiv  Detail & Related papers  (2020-10-30T17:28:40Z)
• Distortion-aware Monocular Depth Estimation for Omnidirectional Images [26.027353545874522]
  We propose a Distortion-Aware Monocular Omnidirectional (DAMO) dense depth estimation network to address this challenge on indoor panoramas. First, we introduce a distortion-aware module to extract calibrated semantic features from omnidirectional images. Second, we introduce a plug-and-play spherical-aware weight matrix for our objective function to handle the uneven distribution of areas projected from a sphere.
  arXiv  Detail & Related papers  (2020-10-18T08:47:57Z)
• A Deep Ordinal Distortion Estimation Approach for Distortion Rectification [62.72089758481803]
  We propose a novel distortion rectification approach that can obtain more accurate parameters with higher efficiency. We design a local-global associated estimation network that learns the ordinal distortion to approximate the realistic distortion distribution. Considering the redundancy of distortion information, our approach only uses a part of distorted image for the ordinal distortion estimation.
  arXiv  Detail & Related papers  (2020-07-21T10:03:42Z)
• UnRectDepthNet: Self-Supervised Monocular Depth Estimation using a Generic Framework for Handling Common Camera Distortion Models [8.484676769284578]
  We propose a generic scale-aware self-supervised pipeline for estimating depth, euclidean distance, and visual odometry from unrectified monocular videos. The proposed algorithm is evaluated further on the KITTI rectified dataset, and we achieve state-of-the-art results.
  arXiv  Detail & Related papers  (2020-07-13T20:35:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.