Physics-Informed Ensemble Representation for Light-Field Image Super-Resolution

• URL: http://arxiv.org/abs/2305.20006v1
• Date: Wed, 31 May 2023 16:27:00 GMT
• Title: Physics-Informed Ensemble Representation for Light-Field Image Super-Resolution
• Authors: Manchang Jin, Gaosheng Liu, Kunshu Hu, Xin Luo, Kun Li, Jingyu Yang
• Abstract summary: We analyze the coordinate transformation of the light field (LF) imaging process to reveal the geometric relationship in the LF images. We introduce a new LF subspace of virtual-slit images (VSI) that provide sub-pixel information complementary to sub-aperture images. To super-resolve image structures from undersampled LF data, we propose a geometry-aware decoder, named EPIXformer.
• Score: 12.156009287223382
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent learning-based approaches have achieved significant progress in light field (LF) image super-resolution (SR) by exploring convolution-based or transformer-based network structures. However, LF imaging has many intrinsic physical priors that have not been fully exploited. In this paper, we analyze the coordinate transformation of the LF imaging process to reveal the geometric relationship in the LF images. Based on such geometric priors, we introduce a new LF subspace of virtual-slit images (VSI) that provide sub-pixel information complementary to sub-aperture images. To leverage the abundant correlation across the four-dimensional data with manageable complexity, we propose learning ensemble representation of all $C_4^2$ LF subspaces for more effective feature extraction. To super-resolve image structures from undersampled LF data, we propose a geometry-aware decoder, named EPIXformer, which constrains the transformer's operational searching regions with a LF physical prior. Experimental results on both spatial and angular SR tasks demonstrate that the proposed method outperforms other state-of-the-art schemes, especially in handling various disparities.

Related papers

• LFIC-DRASC: Deep Light Field Image Compression Using Disentangled Representation and Asymmetrical Strip Convolution [51.909036244222904]
  We propose an end-to-end deep LF Image Compression method using Disentangled Representation and Asymmetrical Strip Convolution. Experimental results demonstrate that the proposed LFIC-DRASC achieves an average of 20.5% bit rate reductions.
  arXiv  Detail & Related papers  (2024-09-18T05:33:42Z)
• Enhancing Low-light Light Field Images with A Deep Compensation Unfolding Network [52.77569396659629]
  This paper presents the deep compensation network unfolding (DCUNet) for restoring light field (LF) images captured under low-light conditions. The framework uses the intermediate enhanced result to estimate the illumination map, which is then employed in the unfolding process to produce a new enhanced result. To properly leverage the unique characteristics of LF images, this paper proposes a pseudo-explicit feature interaction module.
  arXiv  Detail & Related papers  (2023-08-10T07:53:06Z)
• Disentangling Light Fields for Super-Resolution and Disparity Estimation [67.50796924758221]
  Light field (LF) cameras record both intensity and directions of light rays, and encode 3D scenes into 4D LF images. It is challenging for convolutional neural networks (CNNs) to process LF images since the spatial and angular information are highly inter-twined with varying disparities. We propose a generic mechanism to disentangle these coupled information for LF image processing.
  arXiv  Detail & Related papers  (2022-02-22T01:04:41Z)
• Detail-Preserving Transformer for Light Field Image Super-Resolution [15.53525700552796]
  We put forth a novel formulation built upon Transformers, by treating light field super-resolution as a sequence-to-sequence reconstruction task. We propose a detail-preserving Transformer (termed as DPT), by leveraging gradient maps of light field to guide the sequence learning. DPT consists of two branches, with each associated with a Transformer for learning from an original or gradient image sequence.
  arXiv  Detail & Related papers  (2022-01-02T12:33:23Z)
• Light Field Image Super-Resolution with Transformers [11.104338786168324]
  CNN-based methods have achieved remarkable performance in LF image SR. We propose a simple but effective Transformer-based method for LF image SR. Our method achieves superior SR performance with a small model size and low computational cost.
  arXiv  Detail & Related papers  (2021-08-17T12:58:11Z)
• Learning Dynamic Interpolation for Extremely Sparse Light Fields with Wide Baselines [42.59723383219793]
  We propose a learnable model, namely dynamic reconstruction, to replace the commonly-used geometry warping operation. Experiments show that the reconstructed LF weights achieve much higher PSNR/SSIM and preserve the LF parallax structure better than state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-08-17T02:20:03Z)
• A Parallel Down-Up Fusion Network for Salient Object Detection in Optical Remote Sensing Images [82.87122287748791]
  We propose a novel Parallel Down-up Fusion network (PDF-Net) for salient object detection in optical remote sensing images (RSIs) It takes full advantage of the in-path low- and high-level features and cross-path multi-resolution features to distinguish diversely scaled salient objects and suppress the cluttered backgrounds. Experiments on the ORSSD dataset demonstrate that the proposed network is superior to the state-of-the-art approaches both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2020-10-02T05:27:57Z)
• Deep Selective Combinatorial Embedding and Consistency Regularization for Light Field Super-resolution [93.95828097088608]
  Light field (LF) images acquired by hand-held devices usually suffer from low spatial resolution. The high-dimensionality characteristic and complex geometrical structure of LF images make the problem more challenging than traditional single-image SR. We propose a novel learning-based LF spatial SR framework to explore the coherence among LF sub-aperture images. Experimental results over both synthetic and real-world LF datasets demonstrate the significant advantage of our approach over state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-09-26T08:34:37Z)
• Light Field Spatial Super-resolution via Deep Combinatorial Geometry Embedding and Structural Consistency Regularization [99.96632216070718]
  Light field (LF) images acquired by hand-held devices usually suffer from low spatial resolution. The high-dimensional spatiality characteristic and complex geometrical structure of LF images make the problem more challenging than traditional single-image SR. We propose a novel learning-based LF framework, in which each view of an LF image is first individually super-resolved.
  arXiv  Detail & Related papers  (2020-04-05T14:39:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.