OAFuser: Towards Omni-Aperture Fusion for Light Field Semantic
Segmentation
- URL: http://arxiv.org/abs/2307.15588v2
- Date: Thu, 21 Dec 2023 09:47:19 GMT
- Title: OAFuser: Towards Omni-Aperture Fusion for Light Field Semantic
Segmentation
- Authors: Fei Teng, Jiaming Zhang, Kunyu Peng, Yaonan Wang, Rainer Stiefelhagen,
Kailun Yang
- Abstract summary: We propose a new paradigm, Omni-Aperture Fusion model (OAFuser) for light field cameras.
OAFuser discovers the angular information from sub-aperture images to generate a semantically consistent result.
Our proposed OAFuser achieves state-of-the-art performance on the UrbanLF-Real and -Syn datasets.
- Score: 51.739401680890325
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Light field cameras, by harnessing the power of micro-lens array, are capable
of capturing intricate angular and spatial details. This allows for acquiring
complex light patterns and details from multiple angles, significantly
enhancing the precision of image semantic segmentation, a critical aspect of
scene interpretation in vision intelligence. However, the extensive angular
information of light field cameras contains a large amount of redundant data,
which is overwhelming for the limited hardware resources of intelligent
vehicles. Besides, inappropriate compression leads to information corruption
and data loss. To excavate representative information, we propose a new
paradigm, Omni-Aperture Fusion model (OAFuser), which leverages dense context
from the central view and discovers the angular information from sub-aperture
images to generate a semantically consistent result. To avoid feature loss
during network propagation and simultaneously streamline the redundant
information from the light field camera, we present a simple yet very effective
Sub-Aperture Fusion Module (SAFM) to embed sub-aperture images into angular
features without any additional memory cost. Furthermore, to address the
mismatched spatial information across viewpoints, we present a Center Angular
Rectification Module (CARM) to realize feature resorting and prevent feature
occlusion caused by asymmetric information. Our proposed OAFuser achieves
state-of-the-art performance on the UrbanLF-Real and -Syn datasets and sets a
new record of 84.93% in mIoU on the UrbanLF-Real Extended dataset, with a gain
of +4.53%. The source code of OAFuser will be available at
https://github.com/FeiBryantkit/OAFuser.
Related papers
- Spatial-frequency Dual-Domain Feature Fusion Network for Low-Light Remote Sensing Image Enhancement [49.15531684596958]
We propose a Dual-Domain Feature Fusion Network (DFFN) for low-light remote sensing image enhancement.
The first phase learns amplitude information to restore image brightness, and the second phase learns phase information to refine details.
We have constructed two dark light remote sensing datasets to address the current lack of datasets in dark light remote sensing image enhancement.
arXiv Detail & Related papers (2024-04-26T13:21:31Z) - LF Tracy: A Unified Single-Pipeline Approach for Salient Object
Detection in Light Field Cameras [22.288764512594433]
We propose an efficient paradigm to adapt light field data to enhance Salient Object Detection (SOD)
By utilizing only 28.9M parameters, the model achieves a 10% increase in accuracy with 3M additional parameters compared to its backbone using RGB images and an 86% rise to its backbone using LF images.
arXiv Detail & Related papers (2024-01-30T03:17:02Z) - Mutual-Guided Dynamic Network for Image Fusion [51.615598671899335]
We propose a novel mutual-guided dynamic network (MGDN) for image fusion, which allows for effective information utilization across different locations and inputs.
Experimental results on five benchmark datasets demonstrate that our proposed method outperforms existing methods on four image fusion tasks.
arXiv Detail & Related papers (2023-08-24T03:50:37Z) - Enhancing Low-light Light Field Images with A Deep Compensation Unfolding Network [52.77569396659629]
This paper presents the deep compensation network unfolding (DCUNet) for restoring light field (LF) images captured under low-light conditions.
The framework uses the intermediate enhanced result to estimate the illumination map, which is then employed in the unfolding process to produce a new enhanced result.
To properly leverage the unique characteristics of LF images, this paper proposes a pseudo-explicit feature interaction module.
arXiv Detail & Related papers (2023-08-10T07:53:06Z) - Embracing Compact and Robust Architectures for Multi-Exposure Image
Fusion [50.598654017728045]
We propose a search-based paradigm, involving self-alignment and detail repletion modules for robust multi-exposure image fusion.
By utilizing scene relighting and deformable convolutions, the self-alignment module can accurately align images despite camera movement.
We realize the state-of-the-art performance in comparison to various competitive schemes, yielding a 4.02% and 29.34% improvement in PSNR for general and misaligned scenarios.
arXiv Detail & Related papers (2023-05-20T17:01:52Z) - Learning-based Spatial and Angular Information Separation for Light
Field Compression [29.827366575505557]
We propose a novel neural network that can separate angular and spatial information of a light field.
The network represents spatial information using spatial kernels shared among all Sub-Aperture Images (SAIs), and angular information using sets of angular kernels for each SAI.
arXiv Detail & Related papers (2023-04-13T08:02:38Z) - Stereo Superpixel Segmentation Via Decoupled Dynamic Spatial-Embedding
Fusion Network [17.05076034398913]
We propose a stereo superpixel segmentation method with a decoupling mechanism of spatial information in this work.
To decouple stereo disparity information and spatial information, the spatial information is temporarily removed before fusing the features of stereo image pairs.
Our method can achieve the state-of-the-art performance on the KITTI2015 and Cityscapes datasets, and also verify the efficiency when applied in salient object detection on NJU2K dataset.
arXiv Detail & Related papers (2022-08-17T08:22:50Z) - Light Field Reconstruction Using Convolutional Network on EPI and
Extended Applications [78.63280020581662]
A novel convolutional neural network (CNN)-based framework is developed for light field reconstruction from a sparse set of views.
We demonstrate the high performance and robustness of the proposed framework compared with state-of-the-art algorithms.
arXiv Detail & Related papers (2021-03-24T08:16:32Z) - Deep Sparse Light Field Refocusing [35.796798137910066]
Current methods require for this purpose a dense field of angle views.
We present a novel implementation of digital refocusing based on sparse angular information using neural networks.
arXiv Detail & Related papers (2020-09-05T18:34:55Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.