Two-Stage is Enough: A Concise Deep Unfolding Reconstruction Network for
Flexible Video Compressive Sensing
- URL: http://arxiv.org/abs/2201.05810v1
- Date: Sat, 15 Jan 2022 09:40:22 GMT
- Title: Two-Stage is Enough: A Concise Deep Unfolding Reconstruction Network for
Flexible Video Compressive Sensing
- Authors: Siming Zheng, Xiaoyu Yang, Xin Yuan
- Abstract summary: We show that a 2-stage deep unfolding network can lead to the state-of-the-art (SOTA) results in VCS.
We extend the proposed model for color VCS to perform joint reconstruction and demosaicing.
Our network is also flexible to the mask modulation and scale size for color VCS reconstruction so that a single trained network can be applied to different hardware systems.
- Score: 7.154417066884072
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We consider the reconstruction problem of video compressive sensing (VCS)
under the deep unfolding/rolling structure. Yet, we aim to build a flexible and
concise model using minimum stages. Different from existing deep unfolding
networks used for inverse problems, where more stages are used for higher
performance but without flexibility to different masks and scales, hereby we
show that a 2-stage deep unfolding network can lead to the state-of-the-art
(SOTA) results (with a 1.7dB gain in PSNR over the single stage model, RevSCI)
in VCS. The proposed method possesses the properties of adaptation to new masks
and ready to scale to large data without any additional training thanks to the
advantages of deep unfolding. Furthermore, we extend the proposed model for
color VCS to perform joint reconstruction and demosaicing. Experimental results
demonstrate that our 2-stage model has also achieved SOTA on color VCS
reconstruction, leading to a >2.3dB gain in PSNR over the previous SOTA
algorithm based on plug-and-play framework, meanwhile speeds up the
reconstruction by >17 times. In addition, we have found that our network is
also flexible to the mask modulation and scale size for color VCS
reconstruction so that a single trained network can be applied to different
hardware systems. The code and models will be released to the public.
Related papers
- Dr$^2$Net: Dynamic Reversible Dual-Residual Networks for Memory-Efficient Finetuning [81.0108753452546]
We propose Dynamic Reversible Dual-Residual Networks, or Dr$2$Net, to finetune a pretrained model with substantially reduced memory consumption.
Dr$2$Net contains two types of residual connections, one maintaining the residual structure in the pretrained models, and the other making the network reversible.
We show that Dr$2$Net can reach comparable performance to conventional finetuning but with significantly less memory usage.
arXiv Detail & Related papers (2024-01-08T18:59:31Z) - Sampling-Priors-Augmented Deep Unfolding Network for Robust Video
Compressive Sensing [1.7372440481022124]
We propose a Sampling-Priors-Augmented Deep Unfolding Network (SPA-DUN) for efficient and robust VCS reconstruction.
Under the optimization-inspired deep unfolding framework, a lightweight and efficient U-net is exploited to downsize the model.
Experiments on both simulation and real datasets demonstrate that SPA-DUN is not only applicable for various sampling settings with one single model but also achieves SOTA performance with incredible efficiency.
arXiv Detail & Related papers (2023-07-14T12:05:14Z) - Iterative Soft Shrinkage Learning for Efficient Image Super-Resolution [91.3781512926942]
Image super-resolution (SR) has witnessed extensive neural network designs from CNN to transformer architectures.
This work investigates the potential of network pruning for super-resolution iteration to take advantage of off-the-shelf network designs and reduce the underlying computational overhead.
We propose a novel Iterative Soft Shrinkage-Percentage (ISS-P) method by optimizing the sparse structure of a randomly network at each and tweaking unimportant weights with a small amount proportional to the magnitude scale on-the-fly.
arXiv Detail & Related papers (2023-03-16T21:06:13Z) - Structured Sparsity Learning for Efficient Video Super-Resolution [99.1632164448236]
We develop a structured pruning scheme called Structured Sparsity Learning (SSL) according to the properties of video super-resolution (VSR) models.
In SSL, we design pruning schemes for several key components in VSR models, including residual blocks, recurrent networks, and upsampling networks.
arXiv Detail & Related papers (2022-06-15T17:36:04Z) - Residual Multiplicative Filter Networks for Multiscale Reconstruction [24.962697695403037]
We introduce a new coordinate network architecture and training scheme that enables coarse-to-fine optimization with fine-grained control over the frequency support of learned reconstructions.
We demonstrate how these modifications enable multiscale optimization for coarse-to-fine fitting to natural images.
We then evaluate our model on synthetically generated datasets for the the problem of single-particle cryo-EM reconstruction.
arXiv Detail & Related papers (2022-06-01T20:16:28Z) - Dual-view Snapshot Compressive Imaging via Optical Flow Aided Recurrent
Neural Network [14.796204921975733]
Dual-view snapshot compressive imaging (SCI) aims to capture videos from two field-of-views (FoVs) in a single snapshot.
It is challenging for existing model-based decoding algorithms to reconstruct each individual scene.
We propose an optical flow-aided recurrent neural network for dual video SCI systems, which provides high-quality decoding in seconds.
arXiv Detail & Related papers (2021-09-11T14:24:44Z) - A New Backbone for Hyperspectral Image Reconstruction [90.48427561874402]
3D hyperspectral image (HSI) reconstruction refers to inverse process of snapshot compressive imaging.
Proposal is for a Spatial/Spectral Invariant Residual U-Net, namely SSI-ResU-Net.
We show that SSI-ResU-Net achieves competing performance with over 77.3% reduction in terms of floating-point operations.
arXiv Detail & Related papers (2021-08-17T16:20:51Z) - Dual-constrained Deep Semi-Supervised Coupled Factorization Network with
Enriched Prior [80.5637175255349]
We propose a new enriched prior based Dual-constrained Deep Semi-Supervised Coupled Factorization Network, called DS2CF-Net.
To ex-tract hidden deep features, DS2CF-Net is modeled as a deep-structure and geometrical structure-constrained neural network.
Our network can obtain state-of-the-art performance for representation learning and clustering.
arXiv Detail & Related papers (2020-09-08T13:10:21Z) - Deep Adaptive Inference Networks for Single Image Super-Resolution [72.7304455761067]
Single image super-resolution (SISR) has witnessed tremendous progress in recent years owing to the deployment of deep convolutional neural networks (CNNs)
In this paper, we take a step forward to address this issue by leveraging the adaptive inference networks for deep SISR (AdaDSR)
Our AdaDSR involves an SISR model as backbone and a lightweight adapter module which takes image features and resource constraint as input and predicts a map of local network depth.
arXiv Detail & Related papers (2020-04-08T10:08:20Z) - Temporal Convolutional Attention-based Network For Sequence Modeling [13.972755301732656]
We propose an exploratory architecture referred to Temporal Convolutional Attention-based Network (TCAN)
TCAN combines temporal convolutional network and attention mechanism.
We improve the state-of-the-art results of bpc/perplexity to 30.28 on word-level PTB, 1.092 on character-level PTB, and 9.20 on WikiText-2.
arXiv Detail & Related papers (2020-02-28T03:53:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.