FBPT: A Fully Binary Point Transformer

• URL: http://arxiv.org/abs/2403.09998v2
• Date: Thu, 9 May 2024 06:35:38 GMT
• Title: FBPT: A Fully Binary Point Transformer
• Authors: Zhixing Hou, Yuzhang Shang, Yan Yan,
• Abstract summary: This paper presents a novel Fully Binary Point Cloud Transformer (FBPT) model which has the potential to be widely applied and expanded in the fields of robotics and mobile devices. By compressing the weights and activations of a 32-bit full-precision network to 1-bit binary values, the proposed binary point cloud Transformer network significantly reduces the storage footprint and computational resource requirements. The primary focus of this paper is on addressing the performance degradation issue caused by the use of binary point cloud Transformer modules.
• Score: 12.373066597900127
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents a novel Fully Binary Point Cloud Transformer (FBPT) model which has the potential to be widely applied and expanded in the fields of robotics and mobile devices. By compressing the weights and activations of a 32-bit full-precision network to 1-bit binary values, the proposed binary point cloud Transformer network significantly reduces the storage footprint and computational resource requirements of neural network models for point cloud processing tasks, compared to full-precision point cloud networks. However, achieving a fully binary point cloud Transformer network, where all parts except the modules specific to the task are binary, poses challenges and bottlenecks in quantizing the activations of Q, K, V and self-attention in the attention module, as they do not adhere to simple probability distributions and can vary with input data. Furthermore, in our network, the binary attention module undergoes a degradation of the self-attention module due to the uniform distribution that occurs after the softmax operation. The primary focus of this paper is on addressing the performance degradation issue caused by the use of binary point cloud Transformer modules. We propose a novel binarization mechanism called dynamic-static hybridization. Specifically, our approach combines static binarization of the overall network model with fine granularity dynamic binarization of data-sensitive components. Furthermore, we make use of a novel hierarchical training scheme to obtain the optimal model and binarization parameters. These above improvements allow the proposed binarization method to outperform binarization methods applied to convolution neural networks when used in point cloud Transformer structures. To demonstrate the superiority of our algorithm, we conducted experiments on two different tasks: point cloud classification and place recognition.

Related papers

• Tiled Bit Networks: Sub-Bit Neural Network Compression Through Reuse of Learnable Binary Vectors [4.95475852994362]
  We propose a new form of quantization to tile neural network layers with sequences of bits to achieve sub-bit compression of binary-weighted neural networks. We employ the approach to both fully-connected and convolutional layers, which make up the breadth of space in most neural architectures.
  arXiv  Detail & Related papers  (2024-07-16T15:55:38Z)
• Input Layer Binarization with Bit-Plane Encoding [4.872439392746007]
  We present a new method to binarize the first layer using directly the 8-bit representation of input data. The resulting model is fully binarized and our first layer binarization approach is model independent.
  arXiv  Detail & Related papers  (2023-05-04T14:49:07Z)
• BiViT: Extremely Compressed Binary Vision Transformer [19.985314022860432]
  We propose to solve two fundamental challenges to push the horizon of Binary Vision Transformers (BiViT) We propose Softmax-aware Binarization, which dynamically adapts to the data distribution and reduces the error caused by binarization. Our method performs favorably against state-of-the-arts by 19.8% on the TinyImageNet dataset.
  arXiv  Detail & Related papers  (2022-11-14T03:36:38Z)
• BiFSMNv2: Pushing Binary Neural Networks for Keyword Spotting to Real-Network Performance [54.214426436283134]
  Deep neural networks, such as the Deep-FSMN, have been widely studied for keyword spotting (KWS) applications. We present a strong yet efficient binary neural network for KWS, namely BiFSMNv2, pushing it to the real-network accuracy performance. We highlight that benefiting from the compact architecture and optimized hardware kernel, BiFSMNv2 can achieve an impressive 25.1x speedup and 20.2x storage-saving on edge hardware.
  arXiv  Detail & Related papers  (2022-11-13T18:31:45Z)
• CSformer: Bridging Convolution and Transformer for Compressive Sensing [65.22377493627687]
  This paper proposes a hybrid framework that integrates the advantages of leveraging detailed spatial information from CNN and the global context provided by transformer for enhanced representation learning. The proposed approach is an end-to-end compressive image sensing method, composed of adaptive sampling and recovery. The experimental results demonstrate the effectiveness of the dedicated transformer-based architecture for compressive sensing.
  arXiv  Detail & Related papers  (2021-12-31T04:37:11Z)
• Transformers Solve the Limited Receptive Field for Monocular Depth Prediction [82.90445525977904]
  We propose TransDepth, an architecture which benefits from both convolutional neural networks and transformers. This is the first paper which applies transformers into pixel-wise prediction problems involving continuous labels.
  arXiv  Detail & Related papers  (2021-03-22T18:00:13Z)
• BiPointNet: Binary Neural Network for Point Clouds [73.07852523426224]
  BiPointNet is the first model binarization approach for efficient deep learning on point clouds. We show that BiPointNet gives an impressive 14.7x speedup and 18.9x storage saving on real-world resource-constrained devices.
  arXiv  Detail & Related papers  (2020-10-12T07:54:51Z)
• High-Capacity Expert Binary Networks [56.87581500474093]
  Network binarization is a promising hardware-aware direction for creating efficient deep models. Despite its memory and computational advantages, reducing the accuracy gap between binary models and their real-valued counterparts remains an unsolved challenging research problem. We propose Expert Binary Convolution, which, for the first time, tailors conditional computing to binary networks by learning to select one data-specific expert binary filter at a time conditioned on input features.
  arXiv  Detail & Related papers  (2020-10-07T17:58:10Z)
• Binarizing MobileNet via Evolution-based Searching [66.94247681870125]
  We propose a use of evolutionary search to facilitate the construction and training scheme when binarizing MobileNet. Inspired by one-shot architecture search frameworks, we manipulate the idea of group convolution to design efficient 1-Bit Convolutional Neural Networks (CNNs) Our objective is to come up with a tiny yet efficient binary neural architecture by exploring the best candidates of the group convolution.
  arXiv  Detail & Related papers  (2020-05-13T13:25:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.