Learning-based Spatial and Angular Information Separation for Light Field Compression

• URL: http://arxiv.org/abs/2304.06322v4
• Date: Wed, 6 Sep 2023 07:28:33 GMT
• Title: Learning-based Spatial and Angular Information Separation for Light Field Compression
• Authors: Jinglei Shi, Yihong Xu, Christine Guillemot
• Abstract summary: We propose a novel neural network that can separate angular and spatial information of a light field. The network represents spatial information using spatial kernels shared among all Sub-Aperture Images (SAIs), and angular information using sets of angular kernels for each SAI.
• Score: 29.827366575505557
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Light fields are a type of image data that capture both spatial and angular scene information by recording light rays emitted by a scene from different orientations. In this context, spatial information is defined as features that remain static regardless of perspectives, while angular information refers to features that vary between viewpoints. We propose a novel neural network that, by design, can separate angular and spatial information of a light field. The network represents spatial information using spatial kernels shared among all Sub-Aperture Images (SAIs), and angular information using sets of angular kernels for each SAI. To further improve the representation capability of the network without increasing parameter number, we also introduce angular kernel allocation and kernel tensor decomposition mechanisms. Extensive experiments demonstrate the benefits of information separation: when applied to the compression task, our network outperforms other state-of-the-art methods by a large margin. And angular information can be easily transferred to other scenes for rendering dense views, showing the successful separation and the potential use case for the view synthesis task. We plan to release the code upon acceptance of the paper to encourage further research on this topic.

Related papers

• OAFuser: Towards Omni-Aperture Fusion for Light Field Semantic Segmentation [48.828453331724965]
  We propose an Omni-Aperture Fusion model (OAFuser) to extract angular information from sub-aperture images to generate semantically consistent results. The proposed OAFuser achieves state-of-the-art performance on four UrbanLF datasets in terms of all evaluation metrics.
  arXiv  Detail & Related papers  (2023-07-28T14:43:27Z)
• Learning Kernel-Modulated Neural Representation for Efficient Light Field Compression [41.24757573290883]
  We design a compact neural network representation for the light field compression task. It is composed of two types of complementary kernels: descriptive kernels (descriptors) that store scene description information learned during training, and modulatory kernels (modulators) that control the rendering of different SAIs from the queried perspectives.
  arXiv  Detail & Related papers  (2023-07-12T12:58:03Z)
• De-coupling and De-positioning Dense Self-supervised Learning [65.56679416475943]
  Dense Self-Supervised Learning (SSL) methods address the limitations of using image-level feature representations when handling images with multiple objects. We show that they suffer from coupling and positional bias, which arise from the receptive field increasing with layer depth and zero-padding. We demonstrate the benefits of our method on COCO and on a new challenging benchmark, OpenImage-MINI, for object classification, semantic segmentation, and object detection.
  arXiv  Detail & Related papers  (2023-03-29T18:07:25Z)
• SADN: Learned Light Field Image Compression with Spatial-Angular Decorrelation [15.262518233068622]
  We propose a novel end-to-end spatial-angular-decorrelated network (SADN) for high-efficiency light field image compression. The proposed method provides 2.137 times and 2.849 times higher compression efficiency relative to H.266/VVC and H.265/HEVC inter coding. It also outperforms the end-to-end image compression networks by an average of 79.6% saving with much higher subjective quality and light field consistency.
  arXiv  Detail & Related papers  (2022-02-22T11:53:52Z)
• Content-aware Warping for View Synthesis [110.54435867693203]
  We propose content-aware warping, which adaptively learns the weights for pixels of a relatively large neighborhood from their contextual information via a lightweight neural network. Based on this learnable warping module, we propose a new end-to-end learning-based framework for novel view synthesis from two source views. Experimental results on structured light field datasets with wide baselines and unstructured multi-view datasets show that the proposed method significantly outperforms state-of-the-art methods both quantitatively and visually.
  arXiv  Detail & Related papers  (2022-01-22T11:35:05Z)
• Learning Neural Light Fields with Ray-Space Embedding Networks [51.88457861982689]
  We propose a novel neural light field representation that is compact and directly predicts integrated radiance along rays. Our method achieves state-of-the-art quality on dense forward-facing datasets such as the Stanford Light Field dataset.
  arXiv  Detail & Related papers  (2021-12-02T18:59:51Z)
• Light Field Implicit Representation for Flexible Resolution Reconstruction [9.173467982128514]
  We propose an implicit representation model for 4D light fields conditioned on a sparse set of input views. Our model is trained to output the light field values for a continuous range of coordinates. Experiments show that our method achieves state-of-the-art performance for the synthesis of view while being computationally fast.
  arXiv  Detail & Related papers  (2021-11-30T23:59:02Z)
• Light Field Reconstruction Using Convolutional Network on EPI and Extended Applications [78.63280020581662]
  A novel convolutional neural network (CNN)-based framework is developed for light field reconstruction from a sparse set of views. We demonstrate the high performance and robustness of the proposed framework compared with state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2021-03-24T08:16:32Z)
• Angular Luminance for Material Segmentation [6.374538197161135]
  Moving cameras provide multiple intensity measurements per pixel, yet often semantic segmentation, material recognition, and object recognition do not utilize this information. We utilize per-pixel angular luminance distributions as a key feature in discriminating the material of the surface. For real-world materials there is significant intra-class variation that can be managed by building a angular luminance network (AngLNet)
  arXiv  Detail & Related papers  (2020-09-22T21:15:27Z)
• Spatial-Angular Attention Network for Light Field Reconstruction [64.27343801968226]
  We propose a spatial-angular attention network to perceive correspondences in the light field non-locally. Motivated by the non-local attention mechanism, a spatial-angular attention module is introduced to compute the responses from all the positions in the epipolar plane for each pixel in the light field. We then propose a multi-scale reconstruction structure to efficiently implement the non-local attention in the low spatial scale.
  arXiv  Detail & Related papers  (2020-07-05T06:55:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.