Spatio-Temporal Outdoor Lighting Aggregation on Image Sequences using Transformer Networks

• URL: http://arxiv.org/abs/2202.09206v1
• Date: Fri, 18 Feb 2022 14:11:16 GMT
• Title: Spatio-Temporal Outdoor Lighting Aggregation on Image Sequences using Transformer Networks
• Authors: Haebom Lee, Christian Homeyer, Robert Herzog, Jan Rexilius, Carsten Rother
• Abstract summary: In this work, we focus on outdoor lighting estimation by aggregating individual noisy estimates from images. Recent work based on deep neural networks has shown promising results for single image lighting estimation, but suffers from robustness. We tackle this problem by combining lighting estimates from several image views sampled in the angular and temporal domain of an image sequence.
• Score: 23.6427456783115
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we focus on outdoor lighting estimation by aggregating individual noisy estimates from images, exploiting the rich image information from wide-angle cameras and/or temporal image sequences. Photographs inherently encode information about the scene's lighting in the form of shading and shadows. Recovering the lighting is an inverse rendering problem and as that ill-posed. Recent work based on deep neural networks has shown promising results for single image lighting estimation, but suffers from robustness. We tackle this problem by combining lighting estimates from several image views sampled in the angular and temporal domain of an image sequence. For this task, we introduce a transformer architecture that is trained in an end-2-end fashion without any statistical post-processing as required by previous work. Thereby, we propose a positional encoding that takes into account the camera calibration and ego-motion estimation to globally register the individual estimates when computing attention between visual words. We show that our method leads to improved lighting estimation while requiring less hyper-parameters compared to the state-of-the-art.

Related papers

• Exposure Bracketing is All You Need for Unifying Image Restoration and Enhancement Tasks [50.822601495422916]
  We propose to utilize exposure bracketing photography to unify image restoration and enhancement tasks. Due to the difficulty in collecting real-world pairs, we suggest a solution that first pre-trains the model with synthetic paired data. In particular, a temporally modulated recurrent network (TMRNet) and self-supervised adaptation method are proposed.
  arXiv  Detail & Related papers  (2024-01-01T14:14:35Z)
• A Non-Uniform Low-Light Image Enhancement Method with Multi-Scale Attention Transformer and Luminance Consistency Loss [11.585269110131659]
  Low-light image enhancement aims to improve the perception of images collected in dim environments. Existing methods cannot adaptively extract the differentiated luminance information, which will easily cause over-exposure and under-exposure. We propose a multi-scale attention Transformer named MSATr, which sufficiently extracts local and global features for light balance to improve the visual quality.
  arXiv  Detail & Related papers  (2023-12-27T10:07:11Z)
• Personalized Video Relighting With an At-Home Light Stage [0.0]
  We develop a personalized video relighting algorithm that produces high-quality and temporally consistent relit videos in real-time. We show that by just capturing recordings of a user watching YouTube videos on a monitor we can train a personalized algorithm capable of performing high-quality relighting under any condition.
  arXiv  Detail & Related papers  (2023-11-15T10:33:20Z)
• Low-Light Image Enhancement with Illumination-Aware Gamma Correction and Complete Image Modelling Network [69.96295927854042]
  Low-light environments usually lead to less informative large-scale dark areas. We propose to integrate the effectiveness of gamma correction with the strong modelling capacities of deep networks. Because exponential operation introduces high computational complexity, we propose to use Taylor Series to approximate gamma correction.
  arXiv  Detail & Related papers  (2023-08-16T08:46:51Z)
• Neural Radiance Transfer Fields for Relightable Novel-view Synthesis with Global Illumination [63.992213016011235]
  We propose a method for scene relighting under novel views by learning a neural precomputed radiance transfer function. Our method can be solely supervised on a set of real images of the scene under a single unknown lighting condition. Results show that the recovered disentanglement of scene parameters improves significantly over the current state of the art.
  arXiv  Detail & Related papers  (2022-07-27T16:07:48Z)
• When the Sun Goes Down: Repairing Photometric Losses for All-Day Depth Estimation [47.617222712429026]
  We show how to use a combination of three techniques to allow the existing photometric losses to work for both day and nighttime images. First, we introduce a per-pixel neural intensity transformation to compensate for the light changes that occur between successive frames. Second, we predict a per-pixel residual flow map that we use to correct the reprojection correspondences induced by the estimated ego-motion and depth.
  arXiv  Detail & Related papers  (2022-06-28T09:29:55Z)
• Relighting Images in the Wild with a Self-Supervised Siamese Auto-Encoder [62.580345486483886]
  We propose a self-supervised method for image relighting of single view images in the wild. The method is based on an auto-encoder which deconstructs an image into two separate encodings. We train our model on large-scale datasets such as Youtube 8M and CelebA.
  arXiv  Detail & Related papers  (2020-12-11T16:08:50Z)
• Towards Geometry Guided Neural Relighting with Flash Photography [26.511476565209026]
  We propose a framework for image relighting from a single flash photograph with its corresponding depth map using deep learning. We experimentally validate the advantage of our geometry guided approach over state-of-the-art image-based approaches in intrinsic image decomposition and image relighting.
  arXiv  Detail & Related papers  (2020-08-12T08:03:28Z)
• Scene relighting with illumination estimation in the latent space on an encoder-decoder scheme [68.8204255655161]
  In this report we present methods that we tried to achieve that goal. Our models are trained on a rendered dataset of artificial locations with varied scene content, light source location and color temperature. With this dataset, we used a network with illumination estimation component aiming to infer and replace light conditions in the latent space representation of the concerned scenes.
  arXiv  Detail & Related papers  (2020-06-03T15:25:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.