MambaSCI: Efficient Mamba-UNet for Quad-Bayer Patterned Video Snapshot Compressive Imaging

• URL: http://arxiv.org/abs/2410.14214v1
• Date: Fri, 18 Oct 2024 07:02:57 GMT
• Title: MambaSCI: Efficient Mamba-UNet for Quad-Bayer Patterned Video Snapshot Compressive Imaging
• Authors: Zhenghao Pan, Haijin Zeng, Jiezhang Cao, Yongyong Chen, Kai Zhang, Yong Xu,
• Abstract summary: Existing color video SCI reconstruction algorithms are designed based on the traditional Bayer pattern. MambaSCI surpasses state-of-the-art methods with lower computational and memory costs.
• Score: 23.69262715870974
• License:
• Abstract: Color video snapshot compressive imaging (SCI) employs computational imaging techniques to capture multiple sequential video frames in a single Bayer-patterned measurement. With the increasing popularity of quad-Bayer pattern in mainstream smartphone cameras for capturing high-resolution videos, mobile photography has become more accessible to a wider audience. However, existing color video SCI reconstruction algorithms are designed based on the traditional Bayer pattern. When applied to videos captured by quad-Bayer cameras, these algorithms often result in color distortion and ineffective demosaicing, rendering them impractical for primary equipment. To address this challenge, we propose the MambaSCI method, which leverages the Mamba and UNet architectures for efficient reconstruction of quad-Bayer patterned color video SCI. To the best of our knowledge, our work presents the first algorithm for quad-Bayer patterned SCI reconstruction, and also the initial application of the Mamba model to this task. Specifically, we customize Residual-Mamba-Blocks, which residually connect the Spatial-Temporal Mamba (STMamba), Edge-Detail-Reconstruction (EDR) module, and Channel Attention (CA) module. Respectively, STMamba is used to model long-range spatial-temporal dependencies with linear complexity, EDR is for better edge-detail reconstruction, and CA is used to compensate for the missing channel information interaction in Mamba model. Experiments demonstrate that MambaSCI surpasses state-of-the-art methods with lower computational and memory costs. PyTorch style pseudo-code for the core modules is provided in the supplementary materials.

Related papers

• MaskMamba: A Hybrid Mamba-Transformer Model for Masked Image Generation [63.73137438677585]
  MaskMamba is a novel hybrid model that combines Mamba and Transformer architectures. It achieves a remarkable $54.44%$ improvement in inference speed at a resolution of $2048times 2048$ over Transformer.
  arXiv  Detail & Related papers  (2024-09-30T04:28:55Z)
• Distillation-free Scaling of Large SSMs for Images and Videos [27.604572990625144]
  State-space models (SSMs) have introduced a novel context modeling method by integrating state-space techniques into deep learning. Mamba-based architectures are difficult to scale with respect to the number of parameters, which is a major limitation for vision applications. We propose a Mamba-Attention interleaved architecture that enhances scalability, robustness, and performance.
  arXiv  Detail & Related papers  (2024-09-18T10:48:10Z)
• Scalable Autoregressive Image Generation with Mamba [23.027439743155192]
  We introduce AiM, an autoregressive (AR) image generative model based on Mamba architecture. Mamba is a novel state-space model characterized by its exceptional performance for long-sequence modeling with linear time. We provide AiM models in various scales, with parameter counts ranging from 148M to 1.3B.
  arXiv  Detail & Related papers  (2024-08-22T09:27:49Z)
• DemMamba: Alignment-free Raw Video Demoireing with Frequency-assisted Spatio-Temporal Mamba [18.06907326360215]
  We propose an alignment-free Raw video demoireing network with frequency-assisted-temporal Mamba (DemMamba) Our proposed MoiMamba surpasses state-of-the-art approaches by 1.3 dB and delivers a superior visual experience.
  arXiv  Detail & Related papers  (2024-08-20T09:31:03Z)
• VideoMamba: Spatio-Temporal Selective State Space Model [18.310796559944347]
  VideoMamba is a novel adaptation of the pure Mamba architecture, specifically designed for video recognition. VideoMamba is not only resource-efficient but also effective in capturing long-range dependency in videos. Our work highlights the potential of VideoMamba as a powerful tool for video understanding, offering a simple yet effective baseline for future research in video analysis.
  arXiv  Detail & Related papers  (2024-07-11T13:11:21Z)
• MambaVision: A Hybrid Mamba-Transformer Vision Backbone [54.965143338206644]
  We propose a novel hybrid Mamba-Transformer backbone, denoted as MambaVision, which is specifically tailored for vision applications. Our core contribution includes redesigning the Mamba formulation to enhance its capability for efficient modeling of visual features. We conduct a comprehensive ablation study on the feasibility of integrating Vision Transformers (ViT) with Mamba.
  arXiv  Detail & Related papers  (2024-07-10T23:02:45Z)
• ReMamber: Referring Image Segmentation with Mamba Twister [51.291487576255435]
  ReMamber is a novel RIS architecture that integrates the power of Mamba with a multi-modal Mamba Twister block. The Mamba Twister explicitly models image-text interaction, and fuses textual and visual features through its unique channel and spatial twisting mechanism.
  arXiv  Detail & Related papers  (2024-03-26T16:27:37Z)
• Event-guided Multi-patch Network with Self-supervision for Non-uniform Motion Deblurring [113.96237446327795]
  We present a novel self-supervised event-guided deep hierarchical Multi-patch Network to deal with blurry images and videos. We also propose an event-guided architecture to exploit motion cues contained in videos to tackle complex blur in videos. Our MPN achieves the state of the art on the GoPro and VideoDeblurring datasets with a 40x faster runtime compared to current multi-scale methods.
  arXiv  Detail & Related papers  (2023-02-14T15:58:00Z)
• Context-Aware Video Reconstruction for Rolling Shutter Cameras [52.28710992548282]
  In this paper, we propose a context-aware GS video reconstruction architecture. We first estimate the bilateral motion field so that the pixels of the two RS frames are warped to a common GS frame. Then, a refinement scheme is proposed to guide the GS frame synthesis along with bilateral occlusion masks to produce high-fidelity GS video frames.
  arXiv  Detail & Related papers  (2022-05-25T17:05:47Z)
• ARVo: Learning All-Range Volumetric Correspondence for Video Deblurring [92.40655035360729]
  Video deblurring models exploit consecutive frames to remove blurs from camera shakes and object motions. We propose a novel implicit method to learn spatial correspondence among blurry frames in the feature space. Our proposed method is evaluated on the widely-adopted DVD dataset, along with a newly collected High-Frame-Rate (1000 fps) dataset for Video Deblurring.
  arXiv  Detail & Related papers  (2021-03-07T04:33:13Z)
• Memory-Efficient Network for Large-scale Video Compressive Sensing [21.040260603729227]
  Video snapshot imaging (SCI) captures a sequence of video frames in a single shot using a 2D detector. In this paper, we develop a memory-efficient network for large-scale video SCI based on multi-group reversible 3D convolutional neural networks.
  arXiv  Detail & Related papers  (2021-03-04T15:14:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.