NeMo: A Neuron-Level Modularizing-While-Training Approach for Decomposing DNN Models

• URL: http://arxiv.org/abs/2508.11348v1
• Date: Fri, 15 Aug 2025 09:25:40 GMT
• Title: NeMo: A Neuron-Level Modularizing-While-Training Approach for Decomposing DNN Models
• Authors: Xiaohan Bi, Binhang Qi, Hailong Sun, Xiang Gao, Yue Yu, Xiaojun Liang,
• Abstract summary: We propose NeMo, a scalable and generalizable modular training approach for deep neural network (DNN) models.<n>NeMo operates at the neuron level fundamental component common to all DNNs-ensuring applicability to Transformers.<n>We show average gains of 1.72% in module classification accuracy and 58.10% reduction in module size, demonstrating efficacy across both CNN and large-scale Transformer-based models.
• Score: 19.733190038554408
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: With the growing incorporation of deep neural network (DNN) models into modern software systems, the prohibitive construction costs have become a significant challenge. Model reuse has been widely applied to reduce training costs, but indiscriminately reusing entire models may incur significant inference overhead. Consequently, DNN modularization has gained attention, enabling module reuse by decomposing DNN models. The emerging modularizing-while-training (MwT) paradigm, which incorporates modularization into training, outperforms modularizing-after-training approaches. However, existing MwT methods focus on small-scale CNN models at the convolutional kernel level and struggle with diverse DNNs and large-scale models, particularly Transformer-based models. To address these limitations, we propose NeMo, a scalable and generalizable MwT approach. NeMo operates at the neuron level fundamental component common to all DNNs-ensuring applicability to Transformers and various architectures. We design a contrastive learning-based modular training method with an effective composite loss function, enabling scalability to large-scale models. Comprehensive experiments on two Transformer-based models and four CNN models across two classification datasets demonstrate NeMo's superiority over state-of-the-art MwT methods. Results show average gains of 1.72% in module classification accuracy and 58.10% reduction in module size, demonstrating efficacy across both CNN and large-scale Transformer-based models. A case study on open-source projects shows NeMo's potential benefits in practical scenarios, offering a promising approach for scalable and generalizable DNN modularization.

Related papers

• Improving DNN Modularization via Activation-Driven Training [5.4070914322511925]
  MODA promotes inherent modularity within a DNN model by directly regulating the activation outputs of its layers. It accomplishes modularization with 29% less training time. It improves the accuracy of a target class by 12% on average while ensuring minimal impact on the accuracy of other classes.
  arXiv  Detail & Related papers  (2024-11-01T23:07:33Z)
• Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging [111.8456671452411]
  Multi-task learning (MTL) leverages a shared model to accomplish multiple tasks and facilitate knowledge transfer. We propose a Weight-Ensembling Mixture of Experts (WEMoE) method for multi-task model merging. We show that WEMoE and E-WEMoE outperform state-of-the-art (SOTA) model merging methods in terms of MTL performance, generalization, and robustness.
  arXiv  Detail & Related papers  (2024-10-29T07:16:31Z)
• Train Faster, Perform Better: Modular Adaptive Training in Over-Parameterized Models [31.960749305728488]
  We introduce a novel concept dubbed modular neural tangent kernel (mNTK) We show that the quality of a module's learning is tightly associated with its mNTK's principal eigenvalue $lambda_max$. We propose a novel training strategy termed Modular Adaptive Training (MAT) to update those modules with their $lambda_max$ exceeding a dynamic threshold.
  arXiv  Detail & Related papers  (2024-05-13T07:46:48Z)
• Diffusion-Based Neural Network Weights Generation [80.89706112736353]
  D2NWG is a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation. Our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• m2mKD: Module-to-Module Knowledge Distillation for Modular Transformers [27.73393245438193]
  We propose module-to-module knowledge distillation (m2mKD) for transferring knowledge between modules. We evaluate m2mKD on two modular neural architectures: Neural Attentive Circuits (NACs) and Vision Mixture-of-Experts (V-MoE) Applying m2mKD to NACs yields significant improvements in IID accuracy on Tiny-ImageNet and OOD robustness on Tiny-ImageNet-R.
  arXiv  Detail & Related papers  (2024-02-26T04:47:32Z)
• Reusing Convolutional Neural Network Models through Modularization and Composition [22.823870645316397]
  We propose two modularization approaches named CNNSplitter and GradSplitter. CNNSplitter decomposes a trained convolutional neural network (CNN) model into $N$ small reusable modules. The resulting modules can be reused to patch existing CNN models or build new CNN models through composition.
  arXiv  Detail & Related papers  (2023-11-08T03:18:49Z)
• Modularizing while Training: A New Paradigm for Modularizing DNN Models [20.892788625187702]
  We propose a novel approach that incorporates modularization into the model training process, i.e., modularizing-while-training (MwT) The accuracy loss caused by MwT is only 1.13 percentage points, which is 1.76 percentage points less than that of the baseline. The total time cost required for training and modularizing is only 108 minutes, half of the baseline.
  arXiv  Detail & Related papers  (2023-06-15T07:45:43Z)
• Decouple Graph Neural Networks: Train Multiple Simple GNNs Simultaneously Instead of One [60.5818387068983]
  Graph neural networks (GNN) suffer from severe inefficiency. We propose to decouple a multi-layer GNN as multiple simple modules for more efficient training. We show that the proposed framework is highly efficient with reasonable performance.
  arXiv  Detail & Related papers  (2023-04-20T07:21:32Z)
• Neural Attentive Circuits [93.95502541529115]
  We introduce a general purpose, yet modular neural architecture called Neural Attentive Circuits (NACs) NACs learn the parameterization and a sparse connectivity of neural modules without using domain knowledge. NACs achieve an 8x speedup at inference time while losing less than 3% performance.
  arXiv  Detail & Related papers  (2022-10-14T18:00:07Z)
• Neural Network Module Decomposition and Recomposition [35.21448933547118]
  We propose a modularization method that decomposes a deep neural network (DNN) into small modules from a functionality perspective. We demonstrate that the proposed method can decompose and recompose DNNs with high compression ratio and high accuracy.
  arXiv  Detail & Related papers  (2021-12-25T08:36:47Z)
• MoEfication: Conditional Computation of Transformer Models for Efficient Inference [66.56994436947441]
  Transformer-based pre-trained language models can achieve superior performance on most NLP tasks due to large parameter capacity, but also lead to huge computation cost. We explore to accelerate large-model inference by conditional computation based on the sparse activation phenomenon. We propose to transform a large model into its mixture-of-experts (MoE) version with equal model size, namely MoEfication.
  arXiv  Detail & Related papers  (2021-10-05T02:14:38Z)
• Accurate and Lightweight Image Super-Resolution with Model-Guided Deep Unfolding Network [63.69237156340457]
  We present and advocate an explainable approach toward SISR named model-guided deep unfolding network (MoG-DUN) MoG-DUN is accurate (producing fewer aliasing artifacts), computationally efficient (with reduced model parameters), and versatile (capable of handling multiple degradations) The superiority of the proposed MoG-DUN method to existing state-of-theart image methods including RCAN, SRDNF, and SRFBN is substantiated by extensive experiments on several popular datasets and various degradation scenarios.
  arXiv  Detail & Related papers  (2020-09-14T08:23:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.