Towards More Effective and Economic Sparsely-Activated Model

• URL: http://arxiv.org/abs/2110.07431v1
• Date: Thu, 14 Oct 2021 14:58:53 GMT
• Title: Towards More Effective and Economic Sparsely-Activated Model
• Authors: Hao Jiang, Ke Zhan, Jianwei Qu, Yongkang Wu, Zhaoye Fei, Xinyu Zhang, Lei Chen, Zhicheng Dou, Xipeng Qiu, Zikai Guo, Ruofei Lai, Jiawen Wu, Enrui Hu, Yinxia Zhang, Yantao Jia, Fan Yu, Zhao Cao
• Abstract summary: We propose an efficient hierarchical routing mechanism that activates multiple experts in a same device. Our methods shed light on the training of extremely large sparse models and experiments prove that our models can achieve significant performance gain.
• Score: 31.979312090196423
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The sparsely-activated models have achieved great success in natural language processing through large-scale parameters and relatively low computational cost, and gradually become a feasible technique for training and implementing extremely large models. Due to the limit of communication cost, activating multiple experts is hardly affordable during training and inference. Therefore, previous work usually activate just one expert at a time to alleviate additional communication cost. Such routing mechanism limits the upper bound of model performance. In this paper, we first investigate a phenomenon that increasing the number of activated experts can boost the model performance with higher sparse ratio. To increase the number of activated experts without an increase in computational cost, we propose SAM (Switch and Mixture) routing, an efficient hierarchical routing mechanism that activates multiple experts in a same device (GPU). Our methods shed light on the training of extremely large sparse models and experiments prove that our models can achieve significant performance gain with great efficiency improvement.

Related papers

• DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs [70.91804882618243]
  This paper proposes DSMoE, a novel approach that achieves sparsification by partitioning pre-trained FFN layers into computational blocks. We implement adaptive expert routing using sigmoid activation and straight-through estimators, enabling tokens to flexibly access different aspects of model knowledge. Experiments on LLaMA models demonstrate that under equivalent computational constraints, DSMoE achieves superior performance compared to existing pruning and MoE approaches.
  arXiv  Detail & Related papers  (2025-02-18T02:37:26Z)
• Training Language Models to Reason Efficiently [14.390800014819439]
  We use reinforcement learning to train large reasoning models to reason efficiently. Our method incentivizes models to minimize unnecessary computational overhead while maintaining accuracy. Experiments on two open-weight large reasoning models demonstrate significant reductions in inference cost while preserving most of the accuracy.
  arXiv  Detail & Related papers  (2025-02-06T19:18:16Z)
• Majority Kernels: An Approach to Leverage Big Model Dynamics for Efficient Small Model Training [32.154166415680066]
  Methods like distillation, compression or quantization help leverage the highly performant large models to induce smaller performant ones. This paper explores the hypothesis that a single training run can simultaneously train a larger model for performance and derive a smaller model for deployment.
  arXiv  Detail & Related papers  (2024-02-07T17:07:41Z)
• EBJR: Energy-Based Joint Reasoning for Adaptive Inference [10.447353952054492]
  State-of-the-art deep learning models have achieved significant performance levels on various benchmarks. Light-weight architectures, on the other hand, achieve moderate accuracies, but at a much more desirable latency. This paper presents a new method of jointly using the large accurate models together with the small fast ones.
  arXiv  Detail & Related papers  (2021-10-20T02:33:31Z)
• Powerpropagation: A sparsity inducing weight reparameterisation [65.85142037667065]
  We introduce Powerpropagation, a new weight- parameterisation for neural networks that leads to inherently sparse models. Models trained in this manner exhibit similar performance, but have a distribution with markedly higher density at zero, allowing more parameters to be pruned safely. Here, we combine Powerpropagation with a traditional weight-pruning technique as well as recent state-of-the-art sparse-to-sparse algorithms, showing superior performance on the ImageNet benchmark.
  arXiv  Detail & Related papers  (2021-10-01T10:03:57Z)
• Scalable and Efficient MoE Training for Multitask Multilingual Models [55.987536562357086]
  We develop a system capable of scaling MoE models efficiently to trillions of parameters. We also present new training methods to improve MoE sample efficiency and leverage expert pruning strategy to improve time efficiency. A model trained with 10 billion parameters on 50 languages can achieve state-of-the-art performance in Machine Translation (MT) and multilingual natural language generation tasks.
  arXiv  Detail & Related papers  (2021-09-22T00:57:46Z)
• Top-KAST: Top-K Always Sparse Training [50.05611544535801]
  We propose Top-KAST, a method that preserves constant sparsity throughout training. We show that it performs comparably to or better than previous works when training models on the established ImageNet benchmark. In addition to our ImageNet results, we also demonstrate our approach in the domain of language modeling.
  arXiv  Detail & Related papers  (2021-06-07T11:13:05Z)
• Online reinforcement learning with sparse rewards through an active inference capsule [62.997667081978825]
  This paper introduces an active inference agent which minimizes the novel free energy of the expected future. Our model is capable of solving sparse-reward problems with a very high sample efficiency. We also introduce a novel method for approximating the prior model from the reward function, which simplifies the expression of complex objectives.
  arXiv  Detail & Related papers  (2021-06-04T10:03:36Z)
• Exploring Sparse Expert Models and Beyond [51.90860155810848]
  Mixture-of-Experts (MoE) models can achieve promising results with outrageous large amount of parameters but constant computation cost. We propose a simple method called expert prototyping that splits experts into different prototypes and applies $k$ top-$1$ routing. This strategy improves the model quality but maintains constant computational costs, and our further exploration on extremely large-scale models reflects that it is more effective in training larger models.
  arXiv  Detail & Related papers  (2021-05-31T16:12:44Z)
• EfficientPose: Scalable single-person pose estimation [3.325625311163864]
  We propose a novel convolutional neural network architecture, called EfficientPose, for single-person pose estimation. Our top-performing model achieves state-of-the-art accuracy on single-person MPII, with low-complexity ConvNets. Due to its low complexity and efficiency, EfficientPose enables real-world applications on edge devices by limiting the memory footprint and computational cost.
  arXiv  Detail & Related papers  (2020-04-25T16:50:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.