Local-to-Global Panorama Inpainting for Locale-Aware Indoor Lighting Prediction

• URL: http://arxiv.org/abs/2303.10344v1
• Date: Sat, 18 Mar 2023 06:18:49 GMT
• Title: Local-to-Global Panorama Inpainting for Locale-Aware Indoor Lighting Prediction
• Authors: Jiayang Bai, Zhen He, Shan Yang, Jie Guo, Zhenyu Chen, Yan Zhang, Yanwen Guo
• Abstract summary: Predicting panoramic indoor lighting from a single perspective image is a fundamental but highly ill-posed problem in computer vision and graphics. Recent methods mostly rely on convolutional neural networks (CNNs) to fill the missing contents in the warped panorama. We propose a local-to-global strategy for large-scale panorama inpainting.
• Score: 28.180205012351802
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Predicting panoramic indoor lighting from a single perspective image is a fundamental but highly ill-posed problem in computer vision and graphics. To achieve locale-aware and robust prediction, this problem can be decomposed into three sub-tasks: depth-based image warping, panorama inpainting and high-dynamic-range (HDR) reconstruction, among which the success of panorama inpainting plays a key role. Recent methods mostly rely on convolutional neural networks (CNNs) to fill the missing contents in the warped panorama. However, they usually achieve suboptimal performance since the missing contents occupy a very large portion in the panoramic space while CNNs are plagued by limited receptive fields. The spatially-varying distortion in the spherical signals further increases the difficulty for conventional CNNs. To address these issues, we propose a local-to-global strategy for large-scale panorama inpainting. In our method, a depth-guided local inpainting is first applied on the warped panorama to fill small but dense holes. Then, a transformer-based network, dubbed PanoTransformer, is designed to hallucinate reasonable global structures in the large holes. To avoid distortion, we further employ cubemap projection in our design of PanoTransformer. The high-quality panorama recovered at any locale helps us to capture spatially-varying indoor illumination with physically-plausible global structures and fine details.

Related papers

• Scaled 360 layouts: Revisiting non-central panoramas [5.2178708158547025]
  We present a novel approach for 3D layout recovery of indoor environments using single non-central panoramas. We exploit the properties of non-central projection systems in a new geometrical processing to recover the scaled layout.
  arXiv  Detail & Related papers  (2024-02-02T14:55:36Z)
• PERF: Panoramic Neural Radiance Field from a Single Panorama [109.31072618058043]
  PERF is a novel view synthesis framework that trains a panoramic neural radiance field from a single panorama. We propose a novel collaborative RGBD inpainting method and a progressive inpainting-and-erasing method to lift up a 360-degree 2D scene to a 3D scene. Our PERF can be widely used for real-world applications, such as panorama-to-3D, text-to-3D, and 3D scene stylization applications.
  arXiv  Detail & Related papers  (2023-10-25T17:59:01Z)
• Low-Light Image Enhancement with Illumination-Aware Gamma Correction and Complete Image Modelling Network [69.96295927854042]
  Low-light environments usually lead to less informative large-scale dark areas. We propose to integrate the effectiveness of gamma correction with the strong modelling capacities of deep networks. Because exponential operation introduces high computational complexity, we propose to use Taylor Series to approximate gamma correction.
  arXiv  Detail & Related papers  (2023-08-16T08:46:51Z)
• PanoGRF: Generalizable Spherical Radiance Fields for Wide-baseline Panoramas [54.4948540627471]
  We propose PanoGRF, Generalizable Spherical Radiance Fields for Wide-baseline Panoramas. Unlike generalizable radiance fields trained on perspective images, PanoGRF avoids the information loss from panorama-to-perspective conversion. Results on multiple panoramic datasets demonstrate that PanoGRF significantly outperforms state-of-the-art generalizable view synthesis methods.
  arXiv  Detail & Related papers  (2023-06-02T13:35:07Z)
• Panoramic Image-to-Image Translation [37.9486466936501]
  We tackle the challenging task of Panoramic Image-to-Image translation (Pano-I2I) for the first time. This task is difficult due to the geometric distortion of panoramic images and the lack of a panoramic image dataset with diverse conditions, like weather or time. We propose a panoramic distortion-aware I2I model that preserves the structure of the panoramic images while consistently translating their global style referenced from a pinhole image.
  arXiv  Detail & Related papers  (2023-04-11T04:08:58Z)
• Layout-guided Indoor Panorama Inpainting with Plane-aware Normalization [10.721512251097606]
  We present an end-to-end deep learning framework for indoor panoramic image inpainting. We exploit both the global and local context of indoor panorama during the inpainting process. Experimental results show that our work outperforms the current state-of-the-art methods on a public panoramic dataset.
  arXiv  Detail & Related papers  (2023-01-13T15:48:40Z)
• GLPanoDepth: Global-to-Local Panoramic Depth Estimation [18.06592473599777]
  We propose a learning-based method for predicting dense depth values of a scene from a monocular omnidirectional image. We show that cubemap vision transformers have a global receptive field at every stage and can provide globally coherent predictions for spherical signals. This global-to-local strategy allows us to fully exploit useful global and local features in the panorama, achieving state-of-the-art performance in panoramic depth estimation.
  arXiv  Detail & Related papers  (2022-02-06T15:11:58Z)
• Z2P: Instant Rendering of Point Clouds [104.1186026323896]
  We present a technique for rendering point clouds using a neural network. Existing point rendering techniques either use splatting, or first reconstruct a surface mesh that can then be rendered.
  arXiv  Detail & Related papers  (2021-05-30T13:58:24Z)
• Perspective Plane Program Induction from a Single Image [85.28956922100305]
  We study the inverse graphics problem of inferring a holistic representation for natural images. We formulate this problem as jointly finding the camera pose and scene structure that best describe the input image. Our proposed framework, Perspective Plane Program Induction (P3I), combines search-based and gradient-based algorithms to efficiently solve the problem.
  arXiv  Detail & Related papers  (2020-06-25T21:18:58Z)
• Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images [59.906948203578544]
  We introduce a novel learning-based method to reconstruct the high-quality geometry and complex, spatially-varying BRDF of an arbitrary object. We first estimate per-view depth maps using a deep multi-view stereo network. These depth maps are used to coarsely align the different views. We propose a novel multi-view reflectance estimation network architecture.
  arXiv  Detail & Related papers  (2020-03-27T21:28:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.