Learning Non-Local Spatial-Angular Correlation for Light Field Image Super-Resolution

• URL: http://arxiv.org/abs/2302.08058v3
• Date: Fri, 11 Aug 2023 01:25:14 GMT
• Title: Learning Non-Local Spatial-Angular Correlation for Light Field Image Super-Resolution
• Authors: Zhengyu Liang, Yingqian Wang, Longguang Wang, Jungang Yang, Shilin Zhou, Yulan Guo
• Abstract summary: Exploiting spatial-angular correlation is crucial to light field (LF) image super-resolution (SR) We propose a simple yet effective method to learn the non-local spatial-angular correlation for LF image SR. Our method can fully incorporate the information from all angular views while achieving a global receptive field along the epipolar line.
• Score: 36.69391399634076
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Exploiting spatial-angular correlation is crucial to light field (LF) image super-resolution (SR), but is highly challenging due to its non-local property caused by the disparities among LF images. Although many deep neural networks (DNNs) have been developed for LF image SR and achieved continuously improved performance, existing methods cannot well leverage the long-range spatial-angular correlation and thus suffer a significant performance drop when handling scenes with large disparity variations. In this paper, we propose a simple yet effective method to learn the non-local spatial-angular correlation for LF image SR. In our method, we adopt the epipolar plane image (EPI) representation to project the 4D spatial-angular correlation onto multiple 2D EPI planes, and then develop a Transformer network with repetitive self-attention operations to learn the spatial-angular correlation by modeling the dependencies between each pair of EPI pixels. Our method can fully incorporate the information from all angular views while achieving a global receptive field along the epipolar line. We conduct extensive experiments with insightful visualizations to validate the effectiveness of our method. Comparative results on five public datasets show that our method not only achieves state-of-the-art SR performance, but also performs robust to disparity variations. Code is publicly available at https://github.com/ZhengyuLiang24/EPIT.

Related papers

• Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentation [63.15257949821558]
  Referring Remote Sensing Image (RRSIS) is a new challenge that combines computer vision and natural language processing. Traditional Referring Image (RIS) approaches have been impeded by the complex spatial scales and orientations found in aerial imagery. We introduce the Rotated Multi-Scale Interaction Network (RMSIN), an innovative approach designed for the unique demands of RRSIS.
  arXiv  Detail & Related papers  (2023-12-19T08:14:14Z)
• Physics-Informed Ensemble Representation for Light-Field Image Super-Resolution [12.156009287223382]
  We analyze the coordinate transformation of the light field (LF) imaging process to reveal the geometric relationship in the LF images. We introduce a new LF subspace of virtual-slit images (VSI) that provide sub-pixel information complementary to sub-aperture images. To super-resolve image structures from undersampled LF data, we propose a geometry-aware decoder, named EPIXformer.
  arXiv  Detail & Related papers  (2023-05-31T16:27:00Z)
• Towards Effective Image Manipulation Detection with Proposal Contrastive Learning [61.5469708038966]
  We propose Proposal Contrastive Learning (PCL) for effective image manipulation detection. Our PCL consists of a two-stream architecture by extracting two types of global features from RGB and noise views respectively. Our PCL can be easily adapted to unlabeled data in practice, which can reduce manual labeling costs and promote more generalizable features.
  arXiv  Detail & Related papers  (2022-10-16T13:30:13Z)
• Dual-Flow Transformation Network for Deformable Image Registration with Region Consistency Constraint [95.30864269428808]
  Current deep learning (DL)-based image registration approaches learn the spatial transformation from one image to another by leveraging a convolutional neural network. We present a novel dual-flow transformation network with region consistency constraint which maximizes the similarity of ROIs within a pair of images. Experiments on four public 3D MRI datasets show that the proposed method achieves the best registration performance in accuracy and generalization.
  arXiv  Detail & Related papers  (2021-12-04T05:30:44Z)
• Light Field Image Super-Resolution with Transformers [11.104338786168324]
  CNN-based methods have achieved remarkable performance in LF image SR. We propose a simple but effective Transformer-based method for LF image SR. Our method achieves superior SR performance with a small model size and low computational cost.
  arXiv  Detail & Related papers  (2021-08-17T12:58:11Z)
• Learning Dynamic Interpolation for Extremely Sparse Light Fields with Wide Baselines [42.59723383219793]
  We propose a learnable model, namely dynamic reconstruction, to replace the commonly-used geometry warping operation. Experiments show that the reconstructed LF weights achieve much higher PSNR/SSIM and preserve the LF parallax structure better than state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-08-17T02:20:03Z)
• A Parallel Down-Up Fusion Network for Salient Object Detection in Optical Remote Sensing Images [82.87122287748791]
  We propose a novel Parallel Down-up Fusion network (PDF-Net) for salient object detection in optical remote sensing images (RSIs) It takes full advantage of the in-path low- and high-level features and cross-path multi-resolution features to distinguish diversely scaled salient objects and suppress the cluttered backgrounds. Experiments on the ORSSD dataset demonstrate that the proposed network is superior to the state-of-the-art approaches both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2020-10-02T05:27:57Z)
• Deep Selective Combinatorial Embedding and Consistency Regularization for Light Field Super-resolution [93.95828097088608]
  Light field (LF) images acquired by hand-held devices usually suffer from low spatial resolution. The high-dimensionality characteristic and complex geometrical structure of LF images make the problem more challenging than traditional single-image SR. We propose a novel learning-based LF spatial SR framework to explore the coherence among LF sub-aperture images. Experimental results over both synthetic and real-world LF datasets demonstrate the significant advantage of our approach over state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-09-26T08:34:37Z)
• Light Field Spatial Super-resolution via Deep Combinatorial Geometry Embedding and Structural Consistency Regularization [99.96632216070718]
  Light field (LF) images acquired by hand-held devices usually suffer from low spatial resolution. The high-dimensional spatiality characteristic and complex geometrical structure of LF images make the problem more challenging than traditional single-image SR. We propose a novel learning-based LF framework, in which each view of an LF image is first individually super-resolved.
  arXiv  Detail & Related papers  (2020-04-05T14:39:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.