Adaptive Convolutions with Per-pixel Dynamic Filter Atom
- URL: http://arxiv.org/abs/2108.07895v1
- Date: Tue, 17 Aug 2021 22:04:10 GMT
- Title: Adaptive Convolutions with Per-pixel Dynamic Filter Atom
- Authors: Ze Wang, Zichen Miao, Jun Hu, and Qiang Qiu
- Abstract summary: We introduce scalable dynamic convolutions with per-pixel adapted filters.
As plug-and-play replacements to convolutional layers, the introduced adaptive convolutions with per-pixel dynamic atoms enable explicit modeling of intra-image variance.
We present experiments to show that, the proposed method delivers comparable or even better performance across tasks.
- Score: 24.691793951360914
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Applying feature dependent network weights have been proved to be effective
in many fields. However, in practice, restricted by the enormous size of model
parameters and memory footprints, scalable and versatile dynamic convolutions
with per-pixel adapted filters are yet to be fully explored. In this paper, we
address this challenge by decomposing filters, adapted to each spatial
position, over dynamic filter atoms generated by a light-weight network from
local features. Adaptive receptive fields can be supported by further
representing each filter atom over sets of pre-fixed multi-scale bases. As
plug-and-play replacements to convolutional layers, the introduced adaptive
convolutions with per-pixel dynamic atoms enable explicit modeling of
intra-image variance, while avoiding heavy computation, parameters, and memory
cost. Our method preserves the appealing properties of conventional
convolutions as being translation-equivariant and parametrically efficient. We
present experiments to show that, the proposed method delivers comparable or
even better performance across tasks, and are particularly effective on
handling tasks with significant intra-image variance.
Related papers
- OminiControl: Minimal and Universal Control for Diffusion Transformer [68.3243031301164]
OminiControl is a framework that integrates image conditions into pre-trained Diffusion Transformer (DiT) models.
At its core, OminiControl leverages a parameter reuse mechanism, enabling the DiT to encode image conditions using itself as a powerful backbone.
OminiControl addresses a wide range of image conditioning tasks in a unified manner, including subject-driven generation and spatially-aligned conditions.
arXiv Detail & Related papers (2024-11-22T17:55:15Z) - Spatially-Attentive Patch-Hierarchical Network with Adaptive Sampling
for Motion Deblurring [34.751361664891235]
We propose a pixel adaptive and feature attentive design for handling large blur variations across different spatial locations.
We show that our approach performs favorably against the state-of-the-art deblurring algorithms.
arXiv Detail & Related papers (2024-02-09T01:00:09Z) - Decomposed Guided Dynamic Filters for Efficient RGB-Guided Depth
Completion [46.04264366475848]
RGB-guided depth completion aims at predicting dense depth maps from sparse depth measurements and corresponding RGB images.
Guided dynamic filters generate spatially-variant depth-wise separable convolutional filters from RGB features to guide depth features.
We propose to decompose the guided dynamic filters into a spatially-shared component multiplied by content-adaptive adaptors at each spatial location.
arXiv Detail & Related papers (2023-09-05T08:37:58Z) - Tunable Convolutions with Parametric Multi-Loss Optimization [5.658123802733283]
Behavior of neural networks is irremediably determined by the specific loss and data used during training.
It is often desirable to tune the model at inference time based on external factors such as preferences of the user or dynamic characteristics of the data.
This is especially important to balance the perception-distortion trade-off of ill-posed image-to-image translation tasks.
arXiv Detail & Related papers (2023-04-03T11:36:10Z) - Adaptive Single Image Deblurring [43.02281823557039]
We propose an efficient pixel adaptive and feature attentive design for handling large blur variations within and across different images.
We also propose an effective content-aware global-local filtering module that significantly improves the performance.
arXiv Detail & Related papers (2022-01-01T10:10:19Z) - PnP-DETR: Towards Efficient Visual Analysis with Transformers [146.55679348493587]
Recently, DETR pioneered the solution vision tasks with transformers, it directly translates the image feature map into the object result.
Recent transformer-based image recognition model andTT show consistent efficiency gain.
arXiv Detail & Related papers (2021-09-15T01:10:30Z) - Dynamic Convolution for 3D Point Cloud Instance Segmentation [146.7971476424351]
We propose an approach to instance segmentation from 3D point clouds based on dynamic convolution.
We gather homogeneous points that have identical semantic categories and close votes for the geometric centroids.
The proposed approach is proposal-free, and instead exploits a convolution process that adapts to the spatial and semantic characteristics of each instance.
arXiv Detail & Related papers (2021-07-18T09:05:16Z) - Fine-Grained Dynamic Head for Object Detection [68.70628757217939]
We propose a fine-grained dynamic head to conditionally select a pixel-level combination of FPN features from different scales for each instance.
Experiments demonstrate the effectiveness and efficiency of the proposed method on several state-of-the-art detection benchmarks.
arXiv Detail & Related papers (2020-12-07T08:16:32Z) - Hierarchical Dynamic Filtering Network for RGB-D Salient Object
Detection [91.43066633305662]
The main purpose of RGB-D salient object detection (SOD) is how to better integrate and utilize cross-modal fusion information.
In this paper, we explore these issues from a new perspective.
We implement a kind of more flexible and efficient multi-scale cross-modal feature processing.
arXiv Detail & Related papers (2020-07-13T07:59:55Z) - Efficient Spatially Adaptive Convolution and Correlation [11.167305713900074]
We provide a representation-theoretic framework that allows for spatially varying linear transformations to be applied to the filter.
This framework allows for efficient implementation of extended convolution and correlation for transformation groups such as rotation (in 2D and 3D) and scale.
We present applications to pattern matching, image feature description, vector field visualization, and adaptive image filtering.
arXiv Detail & Related papers (2020-06-23T17:41:10Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.