Deep Selective Combinatorial Embedding and Consistency Regularization for Light Field Super-resolution

• URL: http://arxiv.org/abs/2009.12537v2
• Date: Wed, 6 Oct 2021 08:44:09 GMT
• Title: Deep Selective Combinatorial Embedding and Consistency Regularization for Light Field Super-resolution
• Authors: Jing Jin and Junhui Hou and Zhiyu Zhu and Jie Chen and Sam Kwong
• Abstract summary: Light field (LF) images acquired by hand-held devices usually suffer from low spatial resolution. The high-dimensionality characteristic and complex geometrical structure of LF images make the problem more challenging than traditional single-image SR. We propose a novel learning-based LF spatial SR framework to explore the coherence among LF sub-aperture images. Experimental results over both synthetic and real-world LF datasets demonstrate the significant advantage of our approach over state-of-the-art methods.
• Score: 93.95828097088608
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Light field (LF) images acquired by hand-held devices usually suffer from low spatial resolution as the limited detector resolution has to be shared with the angular dimension. LF spatial super-resolution (SR) thus becomes an indispensable part of the LF camera processing pipeline. The high-dimensionality characteristic and complex geometrical structure of LF images make the problem more challenging than traditional single-image SR. The performance of existing methods is still limited as they fail to thoroughly explore the coherence among LF sub-aperture images (SAIs) and are insufficient in accurately preserving the scene's parallax structure. To tackle this challenge, we propose a novel learning-based LF spatial SR framework. Specifically, each SAI of an LF image is first coarsely and individually super-resolved by exploring the complementary information among SAIs with selective combinatorial geometry embedding. To achieve efficient and effective selection of the complementary information, we propose two novel sub-modules conducted hierarchically: the patch selector provides an option of retrieving similar image patches based on offline disparity estimation to handle large-disparity correlations; and the SAI selector adaptively and flexibly selects the most informative SAIs to improve the embedding efficiency. To preserve the parallax structure among the reconstructed SAIs, we subsequently append a consistency regularization network trained over a structure-aware loss function to refine the parallax relationships over the coarse estimation. In addition, we extend the proposed method to irregular LF data. To the best of our knowledge, this is the first learning-based SR method for irregular LF data. Experimental results over both synthetic and real-world LF datasets demonstrate the significant advantage of our approach over state-of-the-art methods.

Related papers

• LFIC-DRASC: Deep Light Field Image Compression Using Disentangled Representation and Asymmetrical Strip Convolution [51.909036244222904]
  We propose an end-to-end deep LF Image Compression method using Disentangled Representation and Asymmetrical Strip Convolution. Experimental results demonstrate that the proposed LFIC-DRASC achieves an average of 20.5% bit rate reductions.
  arXiv  Detail & Related papers  (2024-09-18T05:33:42Z)
• Diffusion-based Light Field Synthesis [50.24624071354433]
  LFdiff is a diffusion-based generative framework tailored for LF synthesis. We propose DistgUnet, a disentanglement-based noise estimation network, to harness comprehensive LF representations. Extensive experiments demonstrate that LFdiff excels in synthesizing visually pleasing and disparity-controllable light fields.
  arXiv  Detail & Related papers  (2024-02-01T13:13:16Z)
• LFSRDiff: Light Field Image Super-Resolution via Diffusion Models [18.20217829625834]
  Light field (LF) image super-resolution (SR) is a challenging problem due to its inherent ill-posed nature. mainstream LF image SR methods typically adopt a deterministic approach, generating only a single output supervised by pixel-wise loss functions. We introduce LFSRDiff, the first diffusion-based LF image SR model, by incorporating the LF disentanglement mechanism.
  arXiv  Detail & Related papers  (2023-11-27T07:31:12Z)
• Enhancing Low-light Light Field Images with A Deep Compensation Unfolding Network [52.77569396659629]
  This paper presents the deep compensation network unfolding (DCUNet) for restoring light field (LF) images captured under low-light conditions. The framework uses the intermediate enhanced result to estimate the illumination map, which is then employed in the unfolding process to produce a new enhanced result. To properly leverage the unique characteristics of LF images, this paper proposes a pseudo-explicit feature interaction module.
  arXiv  Detail & Related papers  (2023-08-10T07:53:06Z)
• Physics-Informed Ensemble Representation for Light-Field Image Super-Resolution [12.156009287223382]
  We analyze the coordinate transformation of the light field (LF) imaging process to reveal the geometric relationship in the LF images. We introduce a new LF subspace of virtual-slit images (VSI) that provide sub-pixel information complementary to sub-aperture images. To super-resolve image structures from undersampled LF data, we propose a geometry-aware decoder, named EPIXformer.
  arXiv  Detail & Related papers  (2023-05-31T16:27:00Z)
• Light Field Reconstruction via Deep Adaptive Fusion of Hybrid Lenses [67.01164492518481]
  This paper explores the problem of reconstructing high-resolution light field (LF) images from hybrid lenses. We propose a novel end-to-end learning-based approach, which can comprehensively utilize the specific characteristics of the input. Our framework could potentially decrease the cost of high-resolution LF data acquisition and benefit LF data storage and transmission.
  arXiv  Detail & Related papers  (2021-02-14T06:44:47Z)
• Light Field Spatial Super-resolution via Deep Combinatorial Geometry Embedding and Structural Consistency Regularization [99.96632216070718]
  Light field (LF) images acquired by hand-held devices usually suffer from low spatial resolution. The high-dimensional spatiality characteristic and complex geometrical structure of LF images make the problem more challenging than traditional single-image SR. We propose a novel learning-based LF framework, in which each view of an LF image is first individually super-resolved.
  arXiv  Detail & Related papers  (2020-04-05T14:39:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.