LDA-AQU: Adaptive Query-guided Upsampling via Local Deformable Attention

• URL: http://arxiv.org/abs/2411.19585v1
• Date: Fri, 29 Nov 2024 09:59:47 GMT
• Title: LDA-AQU: Adaptive Query-guided Upsampling via Local Deformable Attention
• Authors: Zewen Du, Zhenjiang Hu, Guiyu Zhao, Ying Jin, Hongbin Ma,
• Abstract summary: Feature upsampling is an essential operation in constructing deep convolutional neural networks. In this paper, we find that the local self-attention naturally has the feature guidance capability, and its computational paradigm aligns closely with the essence of feature upsampling. Considering the potential semantic gap between upsampled points and their neighboring points, we introduce the deformation mechanism into the upsampler based on local self-attention. LDA-AQU consistently outperforms previous state-of-the-art upsamplers, achieving performance enhancements of 1.7 AP, 1.5 AP, 2.0 PQ, and 2.5 mIoU compared to the
• Score: 5.652171904017473
• License:
• Abstract: Feature upsampling is an essential operation in constructing deep convolutional neural networks. However, existing upsamplers either lack specific feature guidance or necessitate the utilization of high-resolution feature maps, resulting in a loss of performance and flexibility. In this paper, we find that the local self-attention naturally has the feature guidance capability, and its computational paradigm aligns closely with the essence of feature upsampling (\ie feature reassembly of neighboring points). Therefore, we introduce local self-attention into the upsampling task and demonstrate that the majority of existing upsamplers can be regarded as special cases of upsamplers based on local self-attention. Considering the potential semantic gap between upsampled points and their neighboring points, we further introduce the deformation mechanism into the upsampler based on local self-attention, thereby proposing LDA-AQU. As a novel dynamic kernel-based upsampler, LDA-AQU utilizes the feature of queries to guide the model in adaptively adjusting the position and aggregation weight of neighboring points, thereby meeting the upsampling requirements across various complex scenarios. In addition, LDA-AQU is lightweight and can be easily integrated into various model architectures. We evaluate the effectiveness of LDA-AQU across four dense prediction tasks: object detection, instance segmentation, panoptic segmentation, and semantic segmentation. LDA-AQU consistently outperforms previous state-of-the-art upsamplers, achieving performance enhancements of 1.7 AP, 1.5 AP, 2.0 PQ, and 2.5 mIoU compared to the baseline models in the aforementioned four tasks, respectively. Code is available at \url{https://github.com/duzw9311/LDA-AQU}.

Related papers

• A Refreshed Similarity-based Upsampler for Direct High-Ratio Feature Upsampling [54.05517338122698]
  A popular similarity-based feature upsampling pipeline has been proposed, which utilizes a high-resolution feature as guidance. We propose an explicitly controllable query-key feature alignment from both semantic-aware and detail-aware perspectives. We develop a fine-grained neighbor selection strategy on HR features, which is simple yet effective for alleviating mosaic artifacts.
  arXiv  Detail & Related papers  (2024-07-02T14:12:21Z)
• Progressive Classifier and Feature Extractor Adaptation for Unsupervised Domain Adaptation on Point Clouds [36.596096617244655]
  Unsupervised domain adaptation (UDA) is a critical challenge in the field of point cloud analysis. We propose a novel framework that deeply couples the classifier and feature extractor for 3D UDA. Our PCFEA conducts 3D UDA from two distinct perspectives: macro and micro levels.
  arXiv  Detail & Related papers  (2023-11-27T07:33:15Z)
• Small Object Detection via Coarse-to-fine Proposal Generation and Imitation Learning [52.06176253457522]
  We propose a two-stage framework tailored for small object detection based on the Coarse-to-fine pipeline and Feature Imitation learning. CFINet achieves state-of-the-art performance on the large-scale small object detection benchmarks, SODA-D and SODA-A.
  arXiv  Detail & Related papers  (2023-08-18T13:13:09Z)
• BIMS-PU: Bi-Directional and Multi-Scale Point Cloud Upsampling [60.257912103351394]
  We develop a new point cloud upsampling pipeline called BIMS-PU. We decompose the up/downsampling procedure into several up/downsampling sub-steps by breaking the target sampling factor into smaller factors. We show that our method achieves superior results to state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-06-25T13:13:37Z)
• SPU-Net: Self-Supervised Point Cloud Upsampling by Coarse-to-Fine Reconstruction with Self-Projection Optimization [52.20602782690776]
  It is expensive and tedious to obtain large scale paired sparse-canned point sets for training from real scanned sparse data. We propose a self-supervised point cloud upsampling network, named SPU-Net, to capture the inherent upsampling patterns of points lying on the underlying object surface. We conduct various experiments on both synthetic and real-scanned datasets, and the results demonstrate that we achieve comparable performance to the state-of-the-art supervised methods.
  arXiv  Detail & Related papers  (2020-12-08T14:14:09Z)
• AFD-Net: Adaptive Fully-Dual Network for Few-Shot Object Detection [8.39479809973967]
  Few-shot object detection (FSOD) aims at learning a detector that can fast adapt to previously unseen objects with scarce examples. Existing methods solve this problem by performing subtasks of classification and localization utilizing a shared component. We present that a general few-shot detector should consider the explicit decomposition of two subtasks, as well as leveraging information from both of them to enhance feature representations.
  arXiv  Detail & Related papers  (2020-11-30T10:21:32Z)
• Learning Affinity-Aware Upsampling for Deep Image Matting [83.02806488958399]
  We show that learning affinity in upsampling provides an effective and efficient approach to exploit pairwise interactions in deep networks. In particular, results on the Composition-1k matting dataset show that A2U achieves a 14% relative improvement in the SAD metric against a strong baseline. Compared with the state-of-the-art matting network, we achieve 8% higher performance with only 40% model complexity.
  arXiv  Detail & Related papers  (2020-11-29T05:09:43Z)
• Controlling for sparsity in sparse factor analysis models: adaptive latent feature sharing for piecewise linear dimensionality reduction [2.896192909215469]
  We propose a simple and tractable parametric feature allocation model which can address key limitations of current latent feature decomposition techniques. We derive a novel adaptive Factor analysis (aFA), as well as, an adaptive probabilistic principle component analysis (aPPCA) capable of flexible structure discovery and dimensionality reduction. We show that aPPCA and aFA can infer interpretable high level features both when applied on raw MNIST and when applied for interpreting autoencoder features.
  arXiv  Detail & Related papers  (2020-06-22T16:09:11Z)
• Self-Guided Adaptation: Progressive Representation Alignment for Domain Adaptive Object Detection [86.69077525494106]
  Unsupervised domain adaptation (UDA) has achieved unprecedented success in improving the cross-domain robustness of object detection models. Existing UDA methods largely ignore the instantaneous data distribution during model learning, which could deteriorate the feature representation given large domain shift. We propose a Self-Guided Adaptation (SGA) model, target at aligning feature representation and transferring object detection models across domains.
  arXiv  Detail & Related papers  (2020-03-19T13:30:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.