High-Frequency Prior-Driven Adaptive Masking for Accelerating Image Super-Resolution

• URL: http://arxiv.org/abs/2505.06975v1
• Date: Sun, 11 May 2025 13:18:03 GMT
• Title: High-Frequency Prior-Driven Adaptive Masking for Accelerating Image Super-Resolution
• Authors: Wei Shang, Dongwei Ren, Wanying Zhang, Pengfei Zhu, Qinghua Hu, Wangmeng Zuo,
• Abstract summary: High-frequency regions are most critical for reconstruction.<n>We propose a training-free adaptive masking module for acceleration.<n>Our method reduces FLOPs by 24--43% for state-of-the-art models.
• Score: 87.56382172827526
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The primary challenge in accelerating image super-resolution lies in reducing computation while maintaining performance and adaptability. Motivated by the observation that high-frequency regions (e.g., edges and textures) are most critical for reconstruction, we propose a training-free adaptive masking module for acceleration that dynamically focuses computation on these challenging areas. Specifically, our method first extracts high-frequency components via Gaussian blur subtraction and adaptively generates binary masks using K-means clustering to identify regions requiring intensive processing. Our method can be easily integrated with both CNNs and Transformers. For CNN-based architectures, we replace standard $3 \times 3$ convolutions with an unfold operation followed by $1 \times 1$ convolutions, enabling pixel-wise sparse computation guided by the mask. For Transformer-based models, we partition the mask into non-overlapping windows and selectively process tokens based on their average values. During inference, unnecessary pixels or windows are pruned, significantly reducing computation. Moreover, our method supports dilation-based mask adjustment to control the processing scope without retraining, and is robust to unseen degradations (e.g., noise, compression). Extensive experiments on benchmarks demonstrate that our method reduces FLOPs by 24--43% for state-of-the-art models (e.g., CARN, SwinIR) while achieving comparable or better quantitative metrics. The source code is available at https://github.com/shangwei5/AMSR

Related papers

• Motion-Aware Adaptive Pixel Pruning for Efficient Local Motion Deblurring [87.56382172827526]
  We propose a trainable mask predictor that identifies blurred regions in the image.<n>We also develop an intra-frame motion analyzer that translates relative pixel displacements into motion trajectories.<n>Our method is trained end-to-end using a combination of reconstruction loss, reblur loss, and mask loss guided by annotated blur masks.
  arXiv  Detail & Related papers  (2025-07-10T12:38:27Z)
• Image Coding for Machines via Feature-Preserving Rate-Distortion Optimization [27.97760974010369]
  We show an approach to reduce the effect of compression on a task loss using the distance between features as a distortion metric.<n>We simplify the RDO formulation to make the distortion term computable using block-based encoders.<n>We show up to 10% bit-rate savings for the same computer vision accuracy compared to RDO based on SSE.
  arXiv  Detail & Related papers  (2025-04-03T02:11:26Z)
• FilterViT and DropoutViT [0.0]
  We introduce an enhanced version of ViT that conducts attention-based QKV operations during the initial stages of downsampling. We propose a filter attention mechanism that uses a Filter Block to create a salient mask for selecting the most informative pixels for attention. This approach effectively decreases the number of tokens involved in the attention, reducing computational complexity and boosting processing speed.
  arXiv  Detail & Related papers  (2024-10-30T05:38:03Z)
• Mask Propagation for Efficient Video Semantic Segmentation [63.09523058489429]
  Video Semantic baseline degradation (VSS) involves assigning a semantic label to each pixel in a video sequence. We propose an efficient mask propagation framework for VSS, called SSSS. Our framework reduces up to 4x FLOPs compared to the per-frame Mask2Former with only up to 2% mIoU on the Cityscapes validation set.
  arXiv  Detail & Related papers  (2023-10-29T09:55:28Z)
• ClusTR: Exploring Efficient Self-attention via Clustering for Vision Transformers [70.76313507550684]
  We propose a content-based sparse attention method, as an alternative to dense self-attention. Specifically, we cluster and then aggregate key and value tokens, as a content-based method of reducing the total token count. The resulting clustered-token sequence retains the semantic diversity of the original signal, but can be processed at a lower computational cost.
  arXiv  Detail & Related papers  (2022-08-28T04:18:27Z)
• Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction [138.04956118993934]
  We propose a novel Transformer-based method, coarse-to-fine sparse Transformer (CST) CST embedding HSI sparsity into deep learning for HSI reconstruction. In particular, CST uses our proposed spectra-aware screening mechanism (SASM) for coarse patch selecting. Then the selected patches are fed into our customized spectra-aggregation hashing multi-head self-attention (SAH-MSA) for fine pixel clustering and self-similarity capturing.
  arXiv  Detail & Related papers  (2022-03-09T16:17:47Z)
• Learning strides in convolutional neural networks [34.20666933112202]
  This work introduces DiffStride, the first downsampling layer with learnable strides. Experiments on audio and image classification show the generality and effectiveness of our solution.
  arXiv  Detail & Related papers  (2022-02-03T16:03:36Z)
• Token Pooling in Vision Transformers [37.11990688046186]
  In vision transformers, self-attention is not the major bottleneck, e.g., more than 80% of the computation is spent on fully-connected layers. We propose a novel token downsampling method, called Token Pooling, efficiently exploiting redundancies in the images and intermediate token representations. Our experiments show that Token Pooling significantly improves the cost-accuracy trade-off over the state-of-the-art downsampling.
  arXiv  Detail & Related papers  (2021-10-08T02:22:50Z)
• Two-Stage Monte Carlo Denoising with Adaptive Sampling and Kernel Pool [4.194950860992213]
  We tackle the problems in Monte Carlo rendering by proposing a two-stage denoiser based on the adaptive sampling strategy. In the first stage, concurrent to adjusting samples per pixel (spp) on-the-fly, we reuse the computations to generate extra denoising kernels applying on the adaptively rendered image. In the second stage, we design the position-aware pooling and semantic alignment operators to improve spatial-temporal stability.
  arXiv  Detail & Related papers  (2021-03-30T07:05:55Z)
• DCT-Mask: Discrete Cosine Transform Mask Representation for Instance Segmentation [50.70679435176346]
  We propose a new mask representation by applying the discrete cosine transform(DCT) to encode the high-resolution binary grid mask into a compact vector. Our method, termed DCT-Mask, could be easily integrated into most pixel-based instance segmentation methods.
  arXiv  Detail & Related papers  (2020-11-19T15:00:21Z)
• LevelSet R-CNN: A Deep Variational Method for Instance Segmentation [79.20048372891935]
  Currently, many state of the art models are based on the Mask R-CNN framework. We propose LevelSet R-CNN, which combines the best of both worlds by obtaining powerful feature representations. We demonstrate the effectiveness of our approach on COCO and Cityscapes datasets.
  arXiv  Detail & Related papers  (2020-07-30T17:52:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.