Hierarchical Separable Video Transformer for Snapshot Compressive Imaging

• URL: http://arxiv.org/abs/2407.11946v2
• Date: Wed, 17 Jul 2024 08:07:58 GMT
• Title: Hierarchical Separable Video Transformer for Snapshot Compressive Imaging
• Authors: Ping Wang, Yulun Zhang, Lishun Wang, Xin Yuan,
• Abstract summary: Hierarchical Separable Video Transformer (HiSViT) is a reconstruction architecture without temporal aggregation. HiSViT is built by multiple groups of Cross-Scale Separable Multi-head Self-Attention (CSS-MSA) and Gated Self-Modulated Feed-Forward Network ( GSM-FFN) Our method outperforms previous methods by $!>!0.5$ with comparable or fewer parameters and complexity.
• Score: 46.23615648331571
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Transformers have achieved the state-of-the-art performance on solving the inverse problem of Snapshot Compressive Imaging (SCI) for video, whose ill-posedness is rooted in the mixed degradation of spatial masking and temporal aliasing. However, previous Transformers lack an insight into the degradation and thus have limited performance and efficiency. In this work, we tailor an efficient reconstruction architecture without temporal aggregation in early layers and Hierarchical Separable Video Transformer (HiSViT) as building block. HiSViT is built by multiple groups of Cross-Scale Separable Multi-head Self-Attention (CSS-MSA) and Gated Self-Modulated Feed-Forward Network (GSM-FFN) with dense connections, each of which is conducted within a separate channel portions at a different scale, for multi-scale interactions and long-range modeling. By separating spatial operations from temporal ones, CSS-MSA introduces an inductive bias of paying more attention within frames instead of between frames while saving computational overheads. GSM-FFN further enhances the locality via gated mechanism and factorized spatial-temporal convolutions. Extensive experiments demonstrate that our method outperforms previous methods by $\!>\!0.5$ dB with comparable or fewer parameters and complexity. The source codes and pretrained models are released at https://github.com/pwangcs/HiSViT.

Related papers

• Look-Around Before You Leap: High-Frequency Injected Transformer for Image Restoration [46.96362010335177]
  In this paper, we propose HIT, a simple yet effective High-frequency Injected Transformer for image restoration. Specifically, we design a window-wise injection module (WIM), which incorporates abundant high-frequency details into the feature map, to provide reliable references for restoring high-quality images. In addition, we introduce a spatial enhancement unit (SEU) to preserve essential spatial relationships that may be lost due to the computations carried out across channel dimensions in the BIM.
  arXiv  Detail & Related papers  (2024-03-30T08:05:00Z)
• Transformer-based Video Saliency Prediction with High Temporal Dimension Decoding [12.595019348741042]
  We propose a transformer-based video saliency prediction approach with high temporal dimension network decoding (THTDNet) This architecture yields comparable performance to multi-branch and over-complicated models on common benchmarks such as DHF1K, UCF-sports and Hollywood-2.
  arXiv  Detail & Related papers  (2024-01-15T20:09:56Z)
• Isomer: Isomerous Transformer for Zero-shot Video Object Segmentation [59.91357714415056]
  We propose two Transformer variants: Context-Sharing Transformer (CST) and Semantic Gathering-Scattering Transformer (S GST) CST learns the global-shared contextual information within image frames with a lightweight computation; S GST models the semantic correlation separately for the foreground and background. Compared with the baseline that uses vanilla Transformers for multi-stage fusion, ours significantly increase the speed by 13 times and achieves new state-of-the-art ZVOS performance.
  arXiv  Detail & Related papers  (2023-08-13T06:12:00Z)
• Block-State Transformers [41.57016890030355]
  State space models (SSMs) have shown impressive results on tasks that require modeling long-range dependencies. We propose a hybrid layer named Block-State Transformer (BST) that internally combines an SSM sublayer for long-range contextualization. We show that our model outperforms similar Transformer-based architectures on language modeling perplexity and generalizes to longer sequences.
  arXiv  Detail & Related papers  (2023-06-15T22:48:08Z)
• FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classification [53.55504611255664]
  FormerTime is a hierarchical representation model for improving the classification capacity for the multivariate time series classification task. It exhibits three aspects of merits: (1) learning hierarchical multi-scale representations from time series data, (2) inheriting the strength of both transformers and convolutional networks, and (3) tacking the efficiency challenges incurred by the self-attention mechanism.
  arXiv  Detail & Related papers  (2023-02-20T07:46:14Z)
• Time-Space Transformers for Video Panoptic Segmentation [3.2489082010225494]
  We propose a solution that simultaneously predicts pixel-level semantic and clip-level instance segmentation. Our network, named VPS-Transformer, combines a convolutional architecture for single-frame panoptic segmentation and a video module based on an instantiation of a pure Transformer block.
  arXiv  Detail & Related papers  (2022-10-07T13:30:11Z)
• Efficient Attention-free Video Shift Transformers [56.87581500474093]
  This paper tackles the problem of efficient video recognition. Video transformers have recently dominated the efficiency (top-1 accuracy vs FLOPs) spectrum. We extend our formulation in the video domain to construct Video Affine-Shift Transformer.
  arXiv  Detail & Related papers  (2022-08-23T17:48:29Z)
• Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging [142.11622043078867]
  We propose a principled Degradation-Aware Unfolding Framework (DAUF) that estimates parameters from the compressed image and physical mask, and then uses these parameters to control each iteration. By plugging HST into DAUF, we establish the first Transformer-based deep unfolding method, Degradation-Aware Unfolding Half-Shuffle Transformer (DAUHST) for HSI reconstruction.
  arXiv  Detail & Related papers  (2022-05-20T11:37:44Z)
• Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction [138.04956118993934]
  We propose a novel Transformer-based method, coarse-to-fine sparse Transformer (CST) CST embedding HSI sparsity into deep learning for HSI reconstruction. In particular, CST uses our proposed spectra-aware screening mechanism (SASM) for coarse patch selecting. Then the selected patches are fed into our customized spectra-aggregation hashing multi-head self-attention (SAH-MSA) for fine pixel clustering and self-similarity capturing.
  arXiv  Detail & Related papers  (2022-03-09T16:17:47Z)
• TransVOS: Video Object Segmentation with Transformers [13.311777431243296]
  We propose a vision transformer to fully exploit and model both the temporal and spatial relationships. To slim the popular two-encoder pipeline, we design a single two-path feature extractor. Experiments demonstrate the superiority of our TransVOS over state-of-the-art methods on both DAVIS and YouTube-VOS datasets.
  arXiv  Detail & Related papers  (2021-06-01T15:56:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.