FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models

• URL: http://arxiv.org/abs/2604.01762v1
• Date: Thu, 02 Apr 2026 08:30:06 GMT
• Title: FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models
• Abstract summary: FourierMoE is a frequency-adaptive router to dispatch tokens to experts specialized in distinct frequency bands.<n>It consistently outperforms competitive baselines in both single-task and multi-task settings.<n>Results highlight the promise of spectral-domain expert adaptation as an effective and parameter-efficient paradigm for LLM fine-tuning.
• Score: 15.447455377430634
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Parameter-efficient fine-tuning (PEFT) has emerged as a crucial paradigm for adapting large language models (LLMs) under constrained computational budgets. However, standard PEFT methods often struggle in multi-task fine-tuning settings, where diverse optimization objectives induce task interference and limited parameter budgets lead to representational deficiency. While recent approaches incorporate mixture-of-experts (MoE) to alleviate these issues, they predominantly operate in the spatial domain, which may introduce structural redundancy and parameter overhead. To overcome these limitations, we reformulate adaptation in the spectral domain. Our spectral analysis reveals that different tasks exhibit distinct frequency energy distributions, and that LLM layers display heterogeneous frequency sensitivities. Motivated by these insights, we propose FourierMoE, which integrates the MoE architecture with the inverse discrete Fourier transform (IDFT) for frequency-aware adaptation. Specifically, FourierMoE employs a frequency-adaptive router to dispatch tokens to experts specialized in distinct frequency bands. Each expert learns a set of conjugate-symmetric complex coefficients, preserving complete phase and amplitude information while theoretically guaranteeing lossless IDFT reconstruction into real-valued spatial weights. Extensive evaluations across 28 benchmarks, multiple model architectures, and scales demonstrate that FourierMoE consistently outperforms competitive baselines in both single-task and multi-task settings while using significantly fewer trainable parameters. These results highlight the promise of spectral-domain expert adaptation as an effective and parameter-efficient paradigm for LLM fine-tuning.

Related papers

• MTL-FNO: A Lightweight Multi-Task Fourier Neural Operator for Sparse Field Reconstruction [15.401658861451809]
  Multi-task Fourier neural operator (MTL-FNO) is an end-to-end joint training framework based on hard parameter sharing.<n> MTL-FNO achieves accuracy comparable to or even surpassing that of standard FNO, while reducing total model size by 76% and 60%, respectively.<n>Results show that MTL-FNO achieves accuracy comparable to or even surpassing that of standard FNO, while reducing total model size by 76% and 60%, respectively.
  arXiv  Detail & Related papers  (2026-05-26T08:57:31Z)
• Expert Pyramid Tuning: Efficient Parameter Fine-Tuning for Expertise-Driven Task Allocation [7.579163774245479]
  Expert Pyramid Tuning (EPT) is a novel architecture that integrates the multi-scale feature pyramid concept from computer vision into the realm of PEFT.<n>EPT significantly outperforms SOTA MoE-LoRA variants.
  arXiv  Detail & Related papers  (2026-03-13T02:23:41Z)
• FAIM: Frequency-Aware Interactive Mamba for Time Series Classification [87.84511960413715]
  Time series classification (TSC) is crucial in numerous real-world applications, such as environmental monitoring, medical diagnosis, and posture recognition.<n>We propose FAIM, a lightweight Frequency-Aware Interactive Mamba model.<n>We show that FAIM consistently outperforms existing state-of-the-art (SOTA) methods, achieving a superior trade-off between accuracy and efficiency.
  arXiv  Detail & Related papers  (2025-11-26T08:36:33Z)
• General Fourier Feature Physics-Informed Extreme Learning Machine (GFF-PIELM) for High-Frequency PDEs [4.652567513453756]
  We propose a general Fourier feature physics-informed extreme learning machine (GFF-PIELM)<n>GFF-PIELM not only retains the high accuracy, efficiency, and simplicity of the PIELM framework but also inherits the ability of FFMs to effectively handle high-frequency problems.<n>We carry out five case studies with a total of ten numerical examples to highlight the feasibility and validity of the proposed GFF-PIELM.
  arXiv  Detail & Related papers  (2025-10-14T08:55:57Z)
• F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learning [28.598598268071587]
  We conduct the first systematic study of PEFT for pre-trained Large Operator Models (LOMs)<n>We observe that the widely used Low-Rank Adaptation (LoRA) yields markedly poorer performance on LOMs than Adapter tuning.<n>Motivated by the robust empirical gains of adapters and by our theoretical characterization of PDE solutions as spectrally sparse, we introduce Frequency-Adaptive Adapter (F-Adapter)
  arXiv  Detail & Related papers  (2025-09-27T08:05:28Z)
• Learnable Adaptive Time-Frequency Representation via Differentiable Short-Time Fourier Transform [11.05158127763157]
  We propose a differentiable formulation of the STFT that enables gradient-based optimization of its parameters.<n>Our approach integrates seamlessly with neural networks, allowing joint optimization of the STFT parameters and network weights.<n>The efficacy of the proposed differentiable STFT in enhancing TFRs and improving performance in downstream tasks is demonstrated.
  arXiv  Detail & Related papers  (2025-06-26T16:24:27Z)
• SpectrumFM: A Foundation Model for Intelligent Spectrum Management [99.08036558911242]
  Existing intelligent spectrum management methods, typically based on small-scale models, suffer from notable limitations in recognition accuracy, convergence speed, and generalization.<n>This paper proposes a novel spectrum foundation model, termed SpectrumFM, establishing a new paradigm for spectrum management.<n>Experiments demonstrate that SpectrumFM achieves superior performance in terms of accuracy, robustness, adaptability, few-shot learning efficiency, and convergence speed.
  arXiv  Detail & Related papers  (2025-05-02T04:06:39Z)
• Adaptive Frequency Filters As Efficient Global Token Mixers [100.27957692579892]
  We show that adaptive frequency filters can serve as efficient global token mixers. We take AFF token mixers as primary neural operators to build a lightweight neural network, dubbed AFFNet.
  arXiv  Detail & Related papers  (2023-07-26T07:42:28Z)
• Incremental Spatial and Spectral Learning of Neural Operators for Solving Large-Scale PDEs [86.35471039808023]
  We introduce the Incremental Fourier Neural Operator (iFNO), which progressively increases the number of frequency modes used by the model. We show that iFNO reduces total training time while maintaining or improving generalization performance across various datasets. Our method demonstrates a 10% lower testing error, using 20% fewer frequency modes compared to the existing Fourier Neural Operator, while also achieving a 30% faster training.
  arXiv  Detail & Related papers  (2022-11-28T09:57:15Z)
• Transform Once: Efficient Operator Learning in Frequency Domain [69.74509540521397]
  We study deep neural networks designed to harness the structure in frequency domain for efficient learning of long-range correlations in space or time. This work introduces a blueprint for frequency domain learning through a single transform: transform once (T1)
  arXiv  Detail & Related papers  (2022-11-26T01:56:05Z)
• FAMLP: A Frequency-Aware MLP-Like Architecture For Domain Generalization [73.41395947275473]
  We propose a novel frequency-aware architecture, in which the domain-specific features are filtered out in the transformed frequency domain. Experiments on three benchmarks demonstrate significant performance, outperforming the state-of-the-art methods by a margin of 3%, 4% and 9%, respectively.
  arXiv  Detail & Related papers  (2022-03-24T07:26:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.