Video Frame Interpolation for Polarization via Swin-Transformer

• URL: http://arxiv.org/abs/2406.11371v1
• Date: Mon, 17 Jun 2024 09:48:54 GMT
• Title: Video Frame Interpolation for Polarization via Swin-Transformer
• Authors: Feng Huang, Xin Zhang, Yixuan Xu, Xuesong Wang, Xianyu Wu,
• Abstract summary: Video Frame Interpolation (VFI) has been extensively explored and demonstrated, yet its application to polarization remains largely unexplored. This study proposes a multi-stage and multi-scale network called Swin-VFI based on the Swin-Transformer. Experimental results demonstrate our approach's superior reconstruction accuracy across all tasks.
• Score: 9.10220649654041
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Video Frame Interpolation (VFI) has been extensively explored and demonstrated, yet its application to polarization remains largely unexplored. Due to the selective transmission of light by polarized filters, longer exposure times are typically required to ensure sufficient light intensity, which consequently lower the temporal sample rates. Furthermore, because polarization reflected by objects varies with shooting perspective, focusing solely on estimating pixel displacement is insufficient to accurately reconstruct the intermediate polarization. To tackle these challenges, this study proposes a multi-stage and multi-scale network called Swin-VFI based on the Swin-Transformer and introduces a tailored loss function to facilitate the network's understanding of polarization changes. To ensure the practicality of our proposed method, this study evaluates its interpolated frames in Shape from Polarization (SfP) and Human Shape Reconstruction tasks, comparing them with other state-of-the-art methods such as CAIN, FLAVR, and VFIT. Experimental results demonstrate our approach's superior reconstruction accuracy across all tasks.

Related papers

• Generalizable Non-Line-of-Sight Imaging with Learnable Physical Priors [52.195637608631955]
  Non-line-of-sight (NLOS) imaging has attracted increasing attention due to its potential applications. Existing NLOS reconstruction approaches are constrained by the reliance on empirical physical priors. We introduce a novel learning-based solution, comprising two key designs: Learnable Path Compensation (LPC) and Adaptive Phasor Field (APF)
  arXiv  Detail & Related papers  (2024-09-21T04:39:45Z)
• Deep Learning Based Speckle Filtering for Polarimetric SAR Images. Application to Sentinel-1 [51.404644401997736]
  We propose a complete framework to remove speckle in polarimetric SAR images using a convolutional neural network. Experiments show that the proposed approach offers exceptional results in both speckle reduction and resolution preservation.
  arXiv  Detail & Related papers  (2024-08-28T10:07:17Z)
• SS-SfP:Neural Inverse Rendering for Self Supervised Shape from (Mixed) Polarization [21.377923666134116]
  Shape from Polarization (SfP) is the problem popularly known as Shape from Polarization (SfP) We present a novel inverse rendering-based framework to estimate the 3D shape (per-pixel surface normals and depth) of objects and scenes from single-view polarization images.
  arXiv  Detail & Related papers  (2024-07-12T14:29:00Z)
• Robust Depth Enhancement via Polarization Prompt Fusion Tuning [112.88371907047396]
  We present a framework that leverages polarization imaging to improve inaccurate depth measurements from various depth sensors. Our method first adopts a learning-based strategy where a neural network is trained to estimate a dense and complete depth map from polarization data and a sensor depth map from different sensors. To further improve the performance, we propose a Polarization Prompt Fusion Tuning (PPFT) strategy to effectively utilize RGB-based models pre-trained on large-scale datasets.
  arXiv  Detail & Related papers  (2024-04-05T17:55:33Z)
• Learning to Deblur Polarized Images [15.415804124776846]
  A polarization camera can capture four polarized images with different polarizer angles in a single shot. The degree of polarization (DoP) and the angle of polarization (AoP) can be directly computed from the captured polarized images. We propose a polarized image deblurring pipeline to solve the problem in a polarization-aware manner.
  arXiv  Detail & Related papers  (2024-02-28T07:56:28Z)
• NeISF: Neural Incident Stokes Field for Geometry and Material Estimation [50.588983686271284]
  Multi-view inverse rendering is the problem of estimating the scene parameters such as shapes, materials, or illuminations from a sequence of images captured under different viewpoints. We propose Neural Incident Stokes Fields (NeISF), a multi-view inverse framework that reduces ambiguities using polarization cues.
  arXiv  Detail & Related papers  (2023-11-22T06:28:30Z)
• PARTNER: Level up the Polar Representation for LiDAR 3D Object Detection [81.16859686137435]
  We present PARTNER, a novel 3D object detector in the polar coordinate. Our method outperforms the previous polar-based works with remarkable margins of 3.68% and 9.15% on and ONCE validation set.
  arXiv  Detail & Related papers  (2023-08-08T01:59:20Z)
• Polarized Color Image Denoising using Pocoformer [42.171036556122644]
  Polarized color photography provides both visual textures and object surficial information in one snapshot. The use of the directional polarizing filter array causes extremely lower photon count and SNR compared to conventional color imaging. We propose a learning-based approach to simultaneously restore clean signals and precise polarization information.
  arXiv  Detail & Related papers  (2022-07-01T05:52:14Z)
• Transparent Shape from a Single View Polarization Image [6.18278691318801]
  This paper presents a learning-based method for transparent surface estimation from a single view polarization image. Existing shape from polarization(SfP) methods have the difficulty in estimating transparent shape since the inherent transmission interference heavily reduces the reliability of physics-based prior.
  arXiv  Detail & Related papers  (2022-04-13T12:24:32Z)
• Shape from Polarization for Complex Scenes in the Wild [93.65746187211958]
  We present a new data-driven approach with physics-based priors to scene-level normal estimation from a single polarization image. We contribute the first real-world scene-level SfP dataset with paired input polarization images and ground-truth normal maps. Our trained model can be generalized to far-field outdoor scenes as the relationship between polarized light and surface normals is not affected by distance.
  arXiv  Detail & Related papers  (2021-12-21T17:30:23Z)
• Monochrome and Color Polarization Demosaicking Using Edge-Aware Residual Interpolation [14.5106375775521]
  A microgrid image polarimeter enables us to acquire a set of polarization images in one shot. Since the polarimeter consists of an image sensor equipped with a monochrome or color polarization filter array, the demosaicking process to interpolate missing pixel values plays a crucial role in obtaining high-quality polarization images. We propose a novel MPFA demosaicking method based on edge-aware residual (EARI) and also extend it to CPFA demosaicking.
  arXiv  Detail & Related papers  (2020-07-28T15:04:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.