Fast and High-Quality Image Denoising via Malleable Convolutions

• URL: http://arxiv.org/abs/2201.00392v2
• Date: Tue, 4 Jan 2022 04:05:01 GMT
• Title: Fast and High-Quality Image Denoising via Malleable Convolutions
• Authors: Yifan Jiang, Bart Wronski, Ben Mildenhall, Jon Barron, Zhangyang Wang, Tianfan Xue
• Abstract summary: We present Malleable Convolution (MalleConv), as an efficient variant of dynamic convolution. Unlike previous works, MalleConv generates a much smaller set of spatially-varying kernels from input. We also build an efficient denoising network using MalleConv, coined as MalleNet.
• Score: 72.18723834537494
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many image processing networks apply a single set of static convolutional kernels across the entire input image, which is sub-optimal for natural images, as they often consist of heterogeneous visual patterns. Recent works in classification, segmentation, and image restoration have demonstrated that dynamic kernels outperform static kernels at modeling local image statistics. However, these works often adopt per-pixel convolution kernels, which introduce high memory and computation costs. To achieve spatial-varying processing without significant overhead, we present Malleable Convolution (MalleConv), as an efficient variant of dynamic convolution. The weights of MalleConv are dynamically produced by an efficient predictor network capable of generating content-dependent outputs at specific spatial locations. Unlike previous works, MalleConv generates a much smaller set of spatially-varying kernels from input, which enlarges the network's receptive field and significantly reduces computational and memory costs. These kernels are then applied to a full-resolution feature map through an efficient slice-and-conv operator with minimum memory overhead. We further build an efficient denoising network using MalleConv, coined as MalleNet. It achieves high quality results without very deep architecture, e.g., reaching 8.91x faster speed compared to the best performed denoising algorithms (SwinIR), while maintaining similar performance. We also show that a single MalleConv added to a standard convolution-based backbone can contribute significantly to reducing the computational cost or boosting image quality at a similar cost. Project page: https://yifanjiang.net/MalleConv.html

Related papers

• Serpent: Scalable and Efficient Image Restoration via Multi-scale Structured State Space Models [22.702352459581434]
  Serpent is an efficient architecture for high-resolution image restoration. We show that Serpent can achieve reconstruction quality on par with state-of-the-art techniques.
  arXiv  Detail & Related papers  (2024-03-26T17:43:15Z)
• KernelWarehouse: Towards Parameter-Efficient Dynamic Convolution [19.021411176761738]
  Dynamic convolution learns a linear mixture of $n$ static kernels weighted with their sample-dependent attentions. Existing designs are parameter-inefficient: they increase the number of convolutional parameters by $n$ times. We propose $ KernelWarehouse, which can strike a favorable trade-off between parameter efficiency and representation power.
  arXiv  Detail & Related papers  (2023-08-16T13:35:09Z)
• T-former: An Efficient Transformer for Image Inpainting [50.43302925662507]
  A class of attention-based network architectures, called transformer, has shown significant performance on natural language processing fields. In this paper, we design a novel attention linearly related to the resolution according to Taylor expansion, and based on this attention, a network called $T$-former is designed for image inpainting. Experiments on several benchmark datasets demonstrate that our proposed method achieves state-of-the-art accuracy while maintaining a relatively low number of parameters and computational complexity.
  arXiv  Detail & Related papers  (2023-05-12T04:10:42Z)
• Accurate Image Restoration with Attention Retractable Transformer [50.05204240159985]
  We propose Attention Retractable Transformer (ART) for image restoration. ART presents both dense and sparse attention modules in the network. We conduct extensive experiments on image super-resolution, denoising, and JPEG compression artifact reduction tasks.
  arXiv  Detail & Related papers  (2022-10-04T07:35:01Z)
• ShuffleMixer: An Efficient ConvNet for Image Super-Resolution [88.86376017828773]
  We propose ShuffleMixer, for lightweight image super-resolution that explores large convolution and channel split-shuffle operation. Specifically, we develop a large depth-wise convolution and two projection layers based on channel splitting and shuffling as the basic component to mix features efficiently. Experimental results demonstrate that the proposed ShuffleMixer is about 6x smaller than the state-of-the-art methods in terms of model parameters and FLOPs.
  arXiv  Detail & Related papers  (2022-05-30T15:26:52Z)
• Content-Aware Convolutional Neural Networks [98.97634685964819]
  Convolutional Neural Networks (CNNs) have achieved great success due to the powerful feature learning ability of convolution layers. We propose a Content-aware Convolution (CAC) that automatically detects the smooth windows and applies a 1x1 convolutional kernel to replace the original large kernel.
  arXiv  Detail & Related papers  (2021-06-30T03:54:35Z)
• Efficient Deep Image Denoising via Class Specific Convolution [24.103826414190216]
  We propose an efficient deep neural network for image denoising based on pixel-wise classification. The proposed method can reduce the computational costs without sacrificing the performance.
  arXiv  Detail & Related papers  (2021-03-02T10:28:15Z)
• Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification [46.885260723836865]
  Deep convolutional neural networks (CNNs) generally improve when fueled with high resolution images. Inspired by the fact that not all regions in an image are task-relevant, we propose a novel framework that performs efficient image classification. Our framework is general and flexible as it is compatible with most of the state-of-the-art light-weighted CNNs.
  arXiv  Detail & Related papers  (2020-10-11T17:55:06Z)
• Locally Masked Convolution for Autoregressive Models [107.4635841204146]
  LMConv is a simple modification to the standard 2D convolution that allows arbitrary masks to be applied to the weights at each location in the image. We learn an ensemble of distribution estimators that share parameters but differ in generation order, achieving improved performance on whole-image density estimation.
  arXiv  Detail & Related papers  (2020-06-22T17:59:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.