Enhanced Event-Based Video Reconstruction with Motion Compensation

• URL: http://arxiv.org/abs/2403.11961v1
• Date: Mon, 18 Mar 2024 16:58:23 GMT
• Title: Enhanced Event-Based Video Reconstruction with Motion Compensation
• Authors: Siying Liu, Pier Luigi Dragotti,
• Abstract summary: We propose warping the input intensity frames and sparse codes to enhance reconstruction quality. A CISTA-Flow network is constructed by integrating a flow network with CISTA-LSTC for motion compensation. Results demonstrate that our approach achieves state-of-the-art reconstruction accuracy and simultaneously provides reliable dense flow estimation.
• Score: 26.03328887451797
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks for event-based video reconstruction often suffer from a lack of interpretability and have high memory demands. A lightweight network called CISTA-LSTC has recently been introduced showing that high-quality reconstruction can be achieved through the systematic design of its architecture. However, its modelling assumption that input signals and output reconstructed frame share the same sparse representation neglects the displacement caused by motion. To address this, we propose warping the input intensity frames and sparse codes to enhance reconstruction quality. A CISTA-Flow network is constructed by integrating a flow network with CISTA-LSTC for motion compensation. The system relies solely on events, in which predicted flow aids in reconstruction and then reconstructed frames are used to facilitate flow estimation. We also introduce an iterative training framework for this combined system. Results demonstrate that our approach achieves state-of-the-art reconstruction accuracy and simultaneously provides reliable dense flow estimation. Furthermore, our model exhibits flexibility in that it can integrate different flow networks, suggesting its potential for further performance enhancement.

Related papers

• ReMatching Dynamic Reconstruction Flow [55.272357926111454]
  We introduce the ReMatching framework, designed to improve generalization quality by incorporating deformation priors into dynamic reconstruction models. The framework is highly adaptable and can be applied to various dynamic representations. Our evaluations on popular benchmarks involving both synthetic and real-world dynamic scenes demonstrate a clear improvement in reconstruction accuracy of current state-of-the-art models.
  arXiv  Detail & Related papers  (2024-11-01T16:09:33Z)
• Simultaneous Map and Object Reconstruction [66.66729715211642]
  We present a method for dynamic surface reconstruction of large-scale urban scenes from LiDAR. We take inspiration from recent novel view synthesis methods and pose the reconstruction problem as a global optimization. By careful modeling of continuous-time motion, our reconstructions can compensate for the rolling shutter effects of rotating LiDAR sensors.
  arXiv  Detail & Related papers  (2024-06-19T23:53:31Z)
• Low-Light Video Enhancement via Spatial-Temporal Consistent Illumination and Reflection Decomposition [68.6707284662443]
  Low-Light Video Enhancement (LLVE) seeks to restore dynamic and static scenes plagued by severe invisibility and noise. One critical aspect is formulating a consistency constraint specifically for temporal-spatial illumination and appearance enhanced versions. We present an innovative video Retinex-based decomposition strategy that operates without the need for explicit supervision.
  arXiv  Detail & Related papers  (2024-05-24T15:56:40Z)
• MsDC-DEQ-Net: Deep Equilibrium Model (DEQ) with Multi-scale Dilated Convolution for Image Compressive Sensing (CS) [0.0]
  Compressive sensing (CS) is a technique that enables the recovery of sparse signals using fewer measurements than traditional sampling methods. We develop an interpretable and concise neural network model for reconstructing natural images using CS. The model, called MsDC-DEQ-Net, exhibits competitive performance compared to state-of-the-art network-based methods.
  arXiv  Detail & Related papers  (2024-01-05T16:25:58Z)
• A Lightweight Recurrent Learning Network for Sustainable Compressed Sensing [27.964167481909588]
  We propose a lightweight but effective deep neural network based on recurrent learning to achieve a sustainable CS system. Our proposed model can achieve a better reconstruction quality than existing state-of-the-art CS algorithms.
  arXiv  Detail & Related papers  (2023-04-23T14:54:15Z)
• Structured Sparsity Learning for Efficient Video Super-Resolution [99.1632164448236]
  We develop a structured pruning scheme called Structured Sparsity Learning (SSL) according to the properties of video super-resolution (VSR) models. In SSL, we design pruning schemes for several key components in VSR models, including residual blocks, recurrent networks, and upsampling networks.
  arXiv  Detail & Related papers  (2022-06-15T17:36:04Z)
• Towards performant and reliable undersampled MR reconstruction via diffusion model sampling [67.73698021297022]
  DiffuseRecon is a novel diffusion model-based MR reconstruction method. It guides the generation process based on the observed signals. It does not require additional training on specific acceleration factors.
  arXiv  Detail & Related papers  (2022-03-08T02:25:38Z)
• Information Prebuilt Recurrent Reconstruction Network for Video Super-Resolution [8.587681982540225]
  The input information received by different recurrent units in the unidirectional recurrent convolutional network is unbalanced. Early reconstruction frames receive less temporal information, resulting in fuzzy or artifact results. We propose an end-to-end information prebuilt recurrent reconstruction network (IPRRN), consisting of an information prebuilt network (IPNet) and a recurrent reconstruction network (RRNet)
  arXiv  Detail & Related papers  (2021-12-10T05:32:23Z)
• Spatio-Temporal Recurrent Networks for Event-Based Optical Flow Estimation [47.984368369734995]
  We introduce a novel recurrent encoding-decoding neural network architecture for event-based optical flow estimation. The network is end-to-end trained with self-supervised learning on the Multi-Vehicle Stereo Event Camera dataset. We have shown that it outperforms all the existing state-of-the-art methods by a large margin.
  arXiv  Detail & Related papers  (2021-09-10T13:37:37Z)
• A Hybrid Video Anomaly Detection Framework via Memory-Augmented Flow Reconstruction and Flow-Guided Frame Prediction [34.565986275769745]
  $textHF2$-VAD is a Hybrid framework that integrates Flow reconstruction and Frame prediction. We design the network of ML-MemAE-SC to memorize normal patterns for optical flow reconstruction. We then employ a Conditional Variational Autoencoder to predict the next frame given several previous frames.
  arXiv  Detail & Related papers  (2021-08-16T01:37:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.