SP$^2$T: Sparse Proxy Attention for Dual-stream Point Transformer

• URL: http://arxiv.org/abs/2412.11540v2
• Date: Fri, 11 Jul 2025 10:38:22 GMT
• Title: SP$^2$T: Sparse Proxy Attention for Dual-stream Point Transformer
• Abstract summary: We propose Sparse Proxy Point Transformer (SP$2$T) -- a local proxy-based dual-stream point transformer with three key innovations.<n>SP$2$T sets state-of-the-art results with acceptable latency on indoor and outdoor 3D comprehension benchmarks.
• Score: 9.162445137828966
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Point transformers have demonstrated remarkable progress in 3D understanding through expanded receptive fields (RF), but further expanding the RF leads to dilution in group attention and decreases detailed feature extraction capability. Proxy, which serves as abstract representations for simplifying feature maps, enables global RF. However, existing proxy-based approaches face critical limitations: Global proxies incur quadratic complexity for large-scale point clouds and suffer positional ambiguity, while local proxy alternatives struggle with 1) Unreliable sampling from the geometrically diverse point cloud, 2) Inefficient proxy interaction computation, and 3) Imbalanced local-global information fusion; To address these challenges, we propose Sparse Proxy Point Transformer (SP$^{2}$T) -- a local proxy-based dual-stream point transformer with three key innovations: First, for reliable sampling, spatial-wise proxy sampling with vertex-based associations enables robust sampling on geometrically diverse point clouds. Second, for efficient proxy interaction, sparse proxy attention with a table-based relative bias effectively achieves the interaction with efficient map-reduce computation. Third, for local-global information fusion, our dual-stream architecture maintains local-global balance through parallel branches. Comprehensive experiments reveal that SP$^{2}$T sets state-of-the-art results with acceptable latency on indoor and outdoor 3D comprehension benchmarks, demonstrating marked improvement (+3.8% mIoU vs. SPoTr@S3DIS, +22.9% mIoU vs. PointASNL@Sem.KITTI) compared to other proxy-based point cloud methods.

Related papers

• The Impact of Federated Learning on Distributed Remote Sensing Archives [0.5833117322405447]
  Earth observation missions have accumulated more than 5 petabytes of imagery, stored and processed across many geographically dispersed platforms.<n>Training machine learning models on such data in a centralized fashion is impractical due to data volume, sovereignty constraints, and geographic distribution.<n> Federated learning (FL) addresses this by keeping data local and exchanging only model updates.
  arXiv  Detail & Related papers  (2026-04-13T14:46:20Z)
• Hierarchical Awareness Adapters with Hybrid Pyramid Feature Fusion for Dense Depth Prediction [9.96771578356719]
  Existing approaches rely on increasingly complex network architectures to regress depth maps.<n>We propose a multilevel conditional random field (CRF) model built upon the Swin Transformer backbone.<n>Our method achieves state-of-the-art performance, reducing Abs Rel to 0.088 ($-$7.4%) and RMSE to 0.316 ($-$5.4%) on NYU Depth v2, while attaining near-perfect threshold accuracy.
  arXiv  Detail & Related papers  (2026-04-03T07:59:26Z)
• Reliable and Private Anonymous Routing for Satellite Constellations [1.9499120576896225]
  This work proposes an enhanced anonymity architecture, evolving the Loopix mix-network, to provide robust security and reliability in volatile topologies.<n>We introduce three primary contributions: A multi-path transport protocol utilizing $(n, k)$ erasure codes, which is demonstrated to counteract the high link volatility and intermittent connectivity that renders standard mix-networks unreliable.<n>We validate this architecture via high-fidelity, packet-level simulations of a LEO constellation.
  arXiv  Detail & Related papers  (2026-02-12T09:43:55Z)
• PUFM++: Point Cloud Upsampling via Enhanced Flow Matching [15.738247394527024]
  PUFM++ is an enhanced flow-matching framework for reconstructing point clouds from sparse, noisy, and partial observations.<n>We introduce a two-stage flow-matching strategy that first learns a direct, straight-path flow from sparse inputs to dense targets, and then refines it using noise-perturbed samples to approximate the terminal marginal distribution better.<n>Experiments on synthetic benchmarks and real-world scans show that PUFM++ sets a new state of the art in point cloud upsampling.
  arXiv  Detail & Related papers  (2025-12-24T06:30:42Z)
• UAGLNet: Uncertainty-Aggregated Global-Local Fusion Network with Cooperative CNN-Transformer for Building Extraction [83.48950950780554]
  Building extraction from remote sensing images is a challenging task due to the complex structure variations of buildings.<n>Existing methods employ convolutional or self-attention blocks to capture the multi-scale features in the segmentation models.<n>We present an Uncertainty-Aggregated Global-Local Fusion Network (UAGLNet) to exploit high-quality global-local visual semantics.
  arXiv  Detail & Related papers  (2025-12-15T02:59:16Z)
• ProxyTransformation: Preshaping Point Cloud Manifold With Proxy Attention For 3D Visual Grounding [47.927810413168345]
  Embodied intelligence requires agents to interact with 3D environments in real time based on language instructions. Existing point cloud enhancement methods often require a tedious process to improve the manifold. We propose Proxy Transformation suitable for multimodal task to efficiently improve the point cloud manifold.
  arXiv  Detail & Related papers  (2025-02-26T15:53:41Z)
• PointCFormer: a Relation-based Progressive Feature Extraction Network for Point Cloud Completion [19.503392612245474]
  Point cloud completion aims to reconstruct the complete 3D shape from incomplete point clouds.<n>We introduce PointCFormer, a transformer framework optimized for robust global retention and precise local detail capture.<n>PointCFormer demonstrates state-of-the-art performance on several widely used benchmarks.
  arXiv  Detail & Related papers  (2024-12-11T14:37:21Z)
• PVAFN: Point-Voxel Attention Fusion Network with Multi-Pooling Enhancing for 3D Object Detection [59.355022416218624]
  integration of point and voxel representations is becoming more common in LiDAR-based 3D object detection. We propose a novel two-stage 3D object detector, called Point-Voxel Attention Fusion Network (PVAFN) PVAFN uses a multi-pooling strategy to integrate both multi-scale and region-specific information effectively.
  arXiv  Detail & Related papers  (2024-08-26T19:43:01Z)
• GSTran: Joint Geometric and Semantic Coherence for Point Cloud Segmentation [33.72549134362884]
  We propose GSTran, a novel transformer network tailored for the segmentation task. The proposed network mainly consists of two principal components: a local geometric transformer and a global semantic transformer. Experiments on ShapeNetPart and S3DIS benchmarks demonstrate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2024-08-21T12:12:37Z)
• Boosting Cross-Domain Point Classification via Distilling Relational Priors from 2D Transformers [59.0181939916084]
  Traditional 3D networks mainly focus on local geometric details and ignore the topological structure between local geometries. We propose a novel Priors Distillation (RPD) method to extract priors from the well-trained transformers on massive images. Experiments on the PointDA-10 and the Sim-to-Real datasets verify that the proposed method consistently achieves the state-of-the-art performance of UDA for point cloud classification.
  arXiv  Detail & Related papers  (2024-07-26T06:29:09Z)
• Double-Shot 3D Shape Measurement with a Dual-Branch Network for Structured Light Projection Profilometry [14.749887303860717]
  We propose a dual-branch Convolutional Neural Network (CNN)-Transformer network (PDCNet) to process different structured light (SL) modalities.<n>Within PDCNet, a Transformer branch is used to capture global perception in the fringe images, while a CNN branch is designed to collect local details in the speckle images.<n>Our method can reduce fringe order ambiguity while producing high-accuracy results on self-made datasets.
  arXiv  Detail & Related papers  (2024-07-19T10:49:26Z)
• SWCF-Net: Similarity-weighted Convolution and Local-global Fusion for Efficient Large-scale Point Cloud Semantic Segmentation [10.328077317786342]
  We propose a Similarity-Weighted Convolution and local-global Fusion Network, named SWCF-Net. Our method achieves a competitive result with less computational cost, and is able to handle large-scale point clouds efficiently.
  arXiv  Detail & Related papers  (2024-06-17T11:54:46Z)
• SparseDet: A Simple and Effective Framework for Fully Sparse LiDAR-based 3D Object Detection [14.137208879448389]
  LiDAR-based sparse 3D object detection plays a crucial role in autonomous driving applications. Existing methods either use the features of a single central voxel as an object proxy, or treat an aggregated cluster of foreground points as an object proxy. We propose SparseDet which designs sparse queries as object proxies.
  arXiv  Detail & Related papers  (2024-06-16T12:11:44Z)
• APPT : Asymmetric Parallel Point Transformer for 3D Point Cloud Understanding [20.87092793669536]
  Transformer-based networks have achieved impressive performance in 3D point cloud understanding. To tackle these problems, we propose Asymmetric Parallel Point Transformer (APPT) APPT is able to capture features globally throughout the entire network while focusing on local-detailed features.
  arXiv  Detail & Related papers  (2023-03-31T06:11:02Z)
• ProxyFormer: Proxy Alignment Assisted Point Cloud Completion with Missing Part Sensitive Transformer [26.027957774725344]
  We propose a novel point cloud completion approach namely ProxyFormer. It divides point clouds into existing (input) and missing (to be predicted) parts and each part communicates information through its proxies. We show that our method outperforms state-of-the-art completion networks on several benchmark datasets.
  arXiv  Detail & Related papers  (2023-02-28T09:25:37Z)
• Flattening-Net: Deep Regular 2D Representation for 3D Point Cloud Analysis [66.49788145564004]
  We present an unsupervised deep neural architecture called Flattening-Net to represent irregular 3D point clouds of arbitrary geometry and topology. Our methods perform favorably against the current state-of-the-art competitors.
  arXiv  Detail & Related papers  (2022-12-17T15:05:25Z)
• SVNet: Where SO(3) Equivariance Meets Binarization on Point Cloud Representation [65.4396959244269]
  The paper tackles the challenge by designing a general framework to construct 3D learning architectures. The proposed approach can be applied to general backbones like PointNet and DGCNN. Experiments on ModelNet40, ShapeNet, and the real-world dataset ScanObjectNN, demonstrated that the method achieves a great trade-off between efficiency, rotation, and accuracy.
  arXiv  Detail & Related papers  (2022-09-13T12:12:19Z)
• CloudAttention: Efficient Multi-Scale Attention Scheme For 3D Point Cloud Learning [81.85951026033787]
  We set transformers in this work and incorporate them into a hierarchical framework for shape classification and part and scene segmentation. We also compute efficient and dynamic global cross attentions by leveraging sampling and grouping at each iteration. The proposed hierarchical model achieves state-of-the-art shape classification in mean accuracy and yields results on par with the previous segmentation methods.
  arXiv  Detail & Related papers  (2022-07-31T21:39:15Z)
• ProxyMix: Proxy-based Mixup Training with Label Refinery for Source-Free Domain Adaptation [73.14508297140652]
  Unsupervised domain adaptation (UDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain. We propose an effective method named Proxy-based Mixup training with label refinery ( ProxyMix) Experiments on three 2D image and one 3D point cloud object recognition benchmarks demonstrate that ProxyMix yields state-of-the-art performance for source-free UDA tasks.
  arXiv  Detail & Related papers  (2022-05-29T03:45:00Z)
• Non-isotropy Regularization for Proxy-based Deep Metric Learning [78.18860829585182]
  We propose non-isotropy regularization ($mathbbNIR$) for proxy-based Deep Metric Learning. This allows us to explicitly induce a non-isotropic distribution of samples around a proxy to optimize for. Experiments highlight consistent generalization benefits of $mathbbNIR$ while achieving competitive and state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-03-16T11:13:20Z)
• Full Transformer Framework for Robust Point Cloud Registration with Deep Information Interaction [9.431484068349903]
  Recent Transformer-based methods have achieved advanced performance in point cloud registration. Recent CNNs fail to model global relations due to their local fields receptive. shallow-wide architecture of Transformers and lack of positional encoding lead to indistinct feature extraction.
  arXiv  Detail & Related papers  (2021-12-17T08:40:52Z)
• DSPoint: Dual-scale Point Cloud Recognition with High-frequency Fusion [17.797795508707864]
  We propose Dual-Scale Point Cloud Recognition with High-frequency Fusion (DSPoint) We reverse the conventional design of applying convolution on voxels and attention to points. Experiments and ablations on widely-adopted ModelNet40, ShapeNet, and S3DIS demonstrate the state-of-the-art performance of our DSPoint.
  arXiv  Detail & Related papers  (2021-11-19T17:25:54Z)
• Conformer: Local Features Coupling Global Representations for Visual Recognition [72.9550481476101]
  We propose a hybrid network structure, termed Conformer, to take advantage of convolutional operations and self-attention mechanisms for enhanced representation learning. Experiments show that Conformer, under the comparable parameter complexity, outperforms the visual transformer (DeiT-B) by 2.3% on ImageNet.
  arXiv  Detail & Related papers  (2021-05-09T10:00:03Z)
• A^2-FPN: Attention Aggregation based Feature Pyramid Network for Instance Segmentation [68.10621089649486]
  We propose Attention Aggregation based Feature Pyramid Network (A2-FPN) to improve multi-scale feature learning. A2-FPN achieves an improvement of 2.0% and 1.4% mask AP when integrated into the strong baselines such as Cascade Mask R-CNN and Hybrid Task Cascade.
  arXiv  Detail & Related papers  (2021-05-07T11:51:08Z)
• Fewer is More: A Deep Graph Metric Learning Perspective Using Fewer Proxies [65.92826041406802]
  We propose a Proxy-based deep Graph Metric Learning approach from the perspective of graph classification. Multiple global proxies are leveraged to collectively approximate the original data points for each class. We design a novel reverse label propagation algorithm, by which the neighbor relationships are adjusted according to ground-truth labels.
  arXiv  Detail & Related papers  (2020-10-26T14:52:42Z)
• BiPointNet: Binary Neural Network for Point Clouds [73.07852523426224]
  BiPointNet is the first model binarization approach for efficient deep learning on point clouds. We show that BiPointNet gives an impressive 14.7x speedup and 18.9x storage saving on real-world resource-constrained devices.
  arXiv  Detail & Related papers  (2020-10-12T07:54:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.