On Point Affiliation in Feature Upsampling
- URL: http://arxiv.org/abs/2307.08198v1
- Date: Mon, 17 Jul 2023 01:59:14 GMT
- Title: On Point Affiliation in Feature Upsampling
- Authors: Wenze Liu, Hao Lu, Yuliang Liu, Zhiguo Cao
- Abstract summary: We introduce the notion of point affiliation into feature upsampling.
We show that an upsampled point can resort to its low-res decoder neighbors and high-res encoder point to reason the affiliation.
This formulation constitutes a novel, lightweight, and universal upsampling solution.
- Score: 32.28512034705838
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We introduce the notion of point affiliation into feature upsampling. By
abstracting a feature map into non-overlapped semantic clusters formed by
points of identical semantic meaning, feature upsampling can be viewed as point
affiliation -- designating a semantic cluster for each upsampled point. In the
framework of kernel-based dynamic upsampling, we show that an upsampled point
can resort to its low-res decoder neighbors and high-res encoder point to
reason the affiliation, conditioned on the mutual similarity between them. We
therefore present a generic formulation for generating similarity-aware
upsampling kernels and prove that such kernels encourage not only semantic
smoothness but also boundary sharpness. This formulation constitutes a novel,
lightweight, and universal upsampling solution, Similarity-Aware Point
Affiliation (SAPA). We show its working mechanism via our preliminary designs
with window-shape kernel. After probing the limitations of the designs on
object detection, we reveal additional insights for upsampling, leading to SAPA
with the dynamic kernel shape. Extensive experiments demonstrate that SAPA
outperforms prior upsamplers and invites consistent performance improvements on
a number of dense prediction tasks, including semantic segmentation, object
detection, instance segmentation, panoptic segmentation, image matting, and
depth estimation. Code is made available at: https://github.com/tiny-smart/sapa
Related papers
- Lidar Panoptic Segmentation and Tracking without Bells and Whistles [48.078270195629415]
We propose a detection-centric network for lidar segmentation and tracking.
One of the core components of our network is the object instance detection branch.
We evaluate our method on several 3D/4D LPS benchmarks and observe that our model establishes a new state-of-the-art among open-sourced models.
arXiv Detail & Related papers (2023-10-19T04:44:43Z) - Learning to Upsample by Learning to Sample [19.849631293898693]
We present DySample, an ultra-lightweight and effective dynamic upsampler.
Compared with former kernel-based dynamic upsamplers, DySample requires no customized package and has much fewer parameters.
DySample outperforms other upsamplers across five dense prediction tasks.
arXiv Detail & Related papers (2023-08-29T07:50:11Z) - Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videos [63.94040814459116]
Self-supervised methods have shown remarkable progress in learning high-level semantics and low-level temporal correspondence.
We propose a novel semantic-aware masked slot attention on top of the fused semantic features and correspondence maps.
We adopt semantic- and instance-level temporal consistency as self-supervision to encourage temporally coherent object-centric representations.
arXiv Detail & Related papers (2023-08-19T09:12:13Z) - PANet: LiDAR Panoptic Segmentation with Sparse Instance Proposal and
Aggregation [15.664835767712775]
This work proposes a new LPS framework named PANet to eliminate the dependency on the offset branch.
PaNet achieves state-of-the-art performance among published works on the Semantic KITII validation and nuScenes validation for the panoptic segmentation task.
arXiv Detail & Related papers (2023-06-27T10:02:28Z) - SAPA: Similarity-Aware Point Affiliation for Feature Upsampling [27.546863377935118]
We introduce point affiliation into feature upsampling, a notion that describes the affiliation of each upsampled point to a semantic cluster formed by local decoder feature points with semantic similarity.
We present a generic formulation for generating upsampling kernels that encourage semantic smoothness and boundary sharpness.
The key idea of our formulation is to generate similarity-aware kernels by comparing the similarity between each encoder feature point and the spatially associated local region of decoder features.
arXiv Detail & Related papers (2022-09-26T17:32:25Z) - BIMS-PU: Bi-Directional and Multi-Scale Point Cloud Upsampling [60.257912103351394]
We develop a new point cloud upsampling pipeline called BIMS-PU.
We decompose the up/downsampling procedure into several up/downsampling sub-steps by breaking the target sampling factor into smaller factors.
We show that our method achieves superior results to state-of-the-art approaches.
arXiv Detail & Related papers (2022-06-25T13:13:37Z) - SASA: Semantics-Augmented Set Abstraction for Point-based 3D Object
Detection [78.90102636266276]
We propose a novel set abstraction method named Semantics-Augmented Set Abstraction (SASA)
Based on the estimated point-wise foreground scores, we then propose a semantics-guided point sampling algorithm to help retain more important foreground points during down-sampling.
In practice, SASA shows to be effective in identifying valuable points related to foreground objects and improving feature learning for point-based 3D detection.
arXiv Detail & Related papers (2022-01-06T08:54:47Z) - Beyond Farthest Point Sampling in Point-Wise Analysis [52.218037492342546]
We propose a novel data-driven sampler learning strategy for point-wise analysis tasks.
We learn sampling and downstream applications jointly.
Our experiments show that jointly learning of the sampler and task brings remarkable improvement over previous baseline methods.
arXiv Detail & Related papers (2021-07-09T08:08:44Z) - SPU-Net: Self-Supervised Point Cloud Upsampling by Coarse-to-Fine
Reconstruction with Self-Projection Optimization [52.20602782690776]
It is expensive and tedious to obtain large scale paired sparse-canned point sets for training from real scanned sparse data.
We propose a self-supervised point cloud upsampling network, named SPU-Net, to capture the inherent upsampling patterns of points lying on the underlying object surface.
We conduct various experiments on both synthetic and real-scanned datasets, and the results demonstrate that we achieve comparable performance to the state-of-the-art supervised methods.
arXiv Detail & Related papers (2020-12-08T14:14:09Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.