SA-MLP: Enhancing Point Cloud Classification with Efficient Addition and Shift Operations in MLP Architectures

• URL: http://arxiv.org/abs/2409.01998v1
• Date: Tue, 3 Sep 2024 15:43:44 GMT
• Title: SA-MLP: Enhancing Point Cloud Classification with Efficient Addition and Shift Operations in MLP Architectures
• Authors: Qiang Zheng, Chao Zhang, Jian Sun,
• Abstract summary: Traditional neural networks heavily rely on multiplication operations, which are computationally expensive. We propose Add-MLP and Shift-MLP, which replace multiplications with addition and shift operations, respectively, significantly enhancing computational efficiency. This study offers an efficient and effective solution for point cloud classification, balancing performance with computational efficiency.
• Score: 46.266960248570086
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This study addresses the computational inefficiencies in point cloud classification by introducing novel MLP-based architectures inspired by recent advances in CNN optimization. Traditional neural networks heavily rely on multiplication operations, which are computationally expensive. To tackle this, we propose Add-MLP and Shift-MLP, which replace multiplications with addition and shift operations, respectively, significantly enhancing computational efficiency. Building on this, we introduce SA-MLP, a hybrid model that intermixes alternately distributed shift and adder layers to replace MLP layers, maintaining the original number of layers without freezing shift layer weights. This design contrasts with the ShiftAddNet model from previous literature, which replaces convolutional layers with shift and adder layers, leading to a doubling of the number of layers and limited representational capacity due to frozen shift weights. Moreover, SA-MLP optimizes learning by setting distinct learning rates and optimizers specifically for the adder and shift layers, fully leveraging their complementary strengths. Extensive experiments demonstrate that while Add-MLP and Shift-MLP achieve competitive performance, SA-MLP significantly surpasses the multiplication-based baseline MLP model and achieves performance comparable to state-of-the-art MLP-based models. This study offers an efficient and effective solution for point cloud classification, balancing performance with computational efficiency.

Related papers

• Transforming Indoor Localization: Advanced Transformer Architecture for NLOS Dominated Wireless Environments with Distributed Sensors [7.630782404476683]
  We introduce a novel tokenization approach, referred to as Sensor Snapshot Tokenization (SST), which preserves variable-specific representations of power delay profile ( PDP) We also propose a lightweight Swish-Gated Linear Unit-based Transformer (L-SwiGLU Transformer) model, designed to reduce computational complexity without compromising localization accuracy.
  arXiv  Detail & Related papers  (2025-01-14T01:16:30Z)
• OP-LoRA: The Blessing of Dimensionality [93.08208871549557]
  Low-rank adapters enable fine-tuning of large models with only a small number of parameters. They often pose optimization challenges, with poor convergence. We introduce an over- parameterized approach that accelerates training without increasing inference costs. We achieve improvements in vision-language tasks and especially notable increases in image generation.
  arXiv  Detail & Related papers  (2024-12-13T18:55:19Z)
• TinyML NLP Approach for Semantic Wireless Sentiment Classification [49.801175302937246]
  We introduce split learning (SL) as an energy-efficient alternative, privacy-preserving tiny machine learning (MLTiny) scheme. Our results show that SL reduces processing power and CO2 emissions while maintaining high accuracy, whereas FL offers a balanced compromise between efficiency and privacy.
  arXiv  Detail & Related papers  (2024-11-09T21:26:59Z)
• Scaling Laws for Predicting Downstream Performance in LLMs [75.28559015477137]
  This work focuses on the pre-training loss as a more-efficient metric for performance estimation. We extend the power law analytical function to predict domain-specific pre-training loss based on FLOPs across data sources. We employ a two-layer neural network to model the non-linear relationship between multiple domain-specific loss and downstream performance.
  arXiv  Detail & Related papers  (2024-10-11T04:57:48Z)
• GERA: Geometric Embedding for Efficient Point Registration Analysis [20.690695788384517]
  We propose a novel point cloud registration network that leverages a pure geometric architecture, constructing geometric information offline. Our method is the first to replace 3D coordinate inputs with offline-constructed geometric encoding, improving generalization and stability.
  arXiv  Detail & Related papers  (2024-10-01T11:19:56Z)
• Resource Allocation for Stable LLM Training in Mobile Edge Computing [11.366306689957353]
  This paper explores a collaborative training framework that integrates mobile users with edge servers to optimize resource allocation. We formulate a multi-objective optimization problem to minimize the total energy consumption and delay during training. We also address the common issue of instability in model performance by incorporating stability enhancements into our objective function.
  arXiv  Detail & Related papers  (2024-09-30T12:36:27Z)
• A Masked Pruning Approach for Dimensionality Reduction in Communication-Efficient Federated Learning Systems [11.639503711252663]
  Federated Learning (FL) represents a growing machine learning (ML) paradigm designed for training models across numerous nodes. We develop a novel algorithm that overcomes limitations by combining a pruning-based method with the FL process. We present an extensive experimental study demonstrating the superior performance of MPFL compared to existing methods.
  arXiv  Detail & Related papers  (2023-12-06T20:29:23Z)
• Boosting Convolution with Efficient MLP-Permutation for Volumetric Medical Image Segmentation [32.645022002807416]
  Multi-layer perceptron (MLP) network has regained popularity among researchers due to their comparable results to ViT. We propose a novel permutable hybrid network for Vol-MedSeg, named PHNet, which capitalizes on the strengths of both convolution neural networks (CNNs) and PHNet.
  arXiv  Detail & Related papers  (2023-03-23T08:59:09Z)
• The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers [59.87030906486969]
  This paper studies the curious phenomenon for machine learning models with Transformer architectures that their activation maps are sparse. We show that sparsity is a prevalent phenomenon that occurs for both natural language processing and vision tasks. We discuss how sparsity immediately implies a way to significantly reduce the FLOP count and improve efficiency for Transformers.
  arXiv  Detail & Related papers  (2022-10-12T15:25:19Z)
• Collaborative Intelligent Reflecting Surface Networks with Multi-Agent Reinforcement Learning [63.83425382922157]
  Intelligent reflecting surface (IRS) is envisioned to be widely applied in future wireless networks. In this paper, we investigate a multi-user communication system assisted by cooperative IRS devices with the capability of energy harvesting.
  arXiv  Detail & Related papers  (2022-03-26T20:37:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.