From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs
Abstract Overview
This paper studies parameter-efficient fine-tuning for mixture-of-experts (MoE) language models and argues that expert-level adaptation is still too coarse because activated experts are internally sparse. The authors propose NSFT, which decomposes each expert into structured channel groups called sub-experts and selects task-relevant groups using both routing importance and intra-expert activation saliency. To make these sparse partial updates effective, the method adds learning-rate scaling and gradient scaling, including a dynamic variant that refreshes importance during training. Experiments on OLMoE-7B and Ling-mini-2.0-16B across domain-specific tasks and general benchmarks evaluate whether this finer adaptation granularity improves the efficiency-performance trade-off.
Novelty
The main novelty is shifting MoE fine-tuning from whole experts to sub-expert channel groups, rather than treating each expert as an indivisible update unit. The work also combines this fine-grained selection with adaptive optimization schemes designed specifically for sparse partial updates inside experts.
Results
Across OLMoE, NSFT achieves the strongest in-domain performance among efficient tuning baselines reported in Table 1, with an average of 46.62 while maintaining a 45.96 general average and using roughly 3.21% to 10.35% trainable parameters depending on the task. On Ling-mini TableQA, NSFT reaches a 34.44 PoT average versus 17.68 for ESFT while using 2.86% trainable parameters, and it also retains stronger general performance than the LoRA and full fine-tuning baselines shown.
Key Points
- NSFT selects trainable sub-experts by grouping expert channels and ranking them with a score that combines expert routing frequency and intra-expert activation saliency.
- The method introduces learning-rate scaling and static or dynamic gradient scaling to compensate for reduced update magnitude and to allocate stronger updates to more important selected groups.
- Empirically, NSFT consistently outperforms expert-level sparse tuning and LoRA baselines on the reported MoE adaptation tasks, while preserving competitive general capability with fewer trainable parameters.