FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts

• URL: http://arxiv.org/abs/2607.00162v1
• Date: Tue, 30 Jun 2026 20:39:55 GMT
• Title: FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts
• Abstract summary: We introduce Fractional-Fourier Mixture of Experts, a mixture-of-experts adapter.<n>Every expert carries a learnable fractional-Fourier order that continuously interpolates between the spatial domain and the Fourier domain.<n>The order is a single scalar per expert, trained with a separate chirp, and the transform is computed with an $mathcalOdlog d)$--FFT surrogate.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Parameter-efficient fine-tuning (PEFT) reparameterizes weight updates in a fixed basis: low-rank adapters operate in the spatial domain, while a recent line of spectral methods operates in a fixed Fourier domain. We argue that the choice of domain is itself a design degree of freedom that should be learned, and that no single basis is optimal across tasks, layers, or tokens. We introduce Fractional-Fourier Mixture of Experts, a mixture-of-experts adapter in which every expert carries a learnable fractional-Fourier order that continuously interpolates between the spatial domain (recovering vanilla LoRA) and the Fourier domain (recovering a spectral adapter). Routing tokens through experts that occupy different points on this spatial-spectral continuum lets the model place each low-rank update in the domain where it is most compact, and -- because fractional-Fourier operators of different orders are mutually incoherent -- makes the experts naturally decorrelated, which reduces interference and improves multi-task composition. The order is a single scalar per expert, trained with a separate optimizer, and the transform is computed with an $\mathcal{O}(d\log d)$ chirp--FFT surrogate, so Fractional-Fourier Mixture of Experts adds negligible cost over standard MoE-LoRA. Across commonsense, mathematical, code, and knowledge benchmarks on LLaMA-3.1-8B and Qwen2.5-7B, Fractional-Fourier Mixture of Experts improves over strong MoE-LoRA and spectral baselines -- including FlyLoRA, FourierMoE, and HMoRA -- while keeping the active-parameter budget small, and analysis shows that the learned orders specialize by task and layer in interpretable ways.

Related papers

• ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs [5.509795962249259]
  We propose ACE (Adapter Consolidation across Experts) to group redundant experts and replace their expert-specific adapters with group-shared higher-rank LoRA modules.<n>Across evaluations covering 12 datasets and four MoE backbones, ACE achieves the highest observed mean accuracy among the parameter-matched PEFT methods.
  arXiv  Detail & Related papers  (2026-09-05T13:05:01Z)
• RAPTOR: Role-Aware Private Training for Mixture-of-Experts [68.5626338219096]
  RAPTOR is a Role-Aware Private Training framework.<n>RAPTOR alternates shared and expert optimization and targets each failure directly.<n> Experiments on Switch Transformer and OLMoE fine-tuning across GLUE tasks, and on DeepSeek-VL2-Tiny, show consistent gains over standard DP baselines.
  arXiv  Detail & Related papers  (2026-09-04T23:16:12Z)
• Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms [50.28963615440865]
  We organize three fusion paradigms by the artefacts they reuse.<n>We compare all three using shared experts and data across model scales and a multi-domain benchmark suite.
  arXiv  Detail & Related papers  (2026-08-27T17:38:04Z)
• Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation [49.8459545291965]
  Mixture-of-experts (MoE) models have recently moved beyond routing a fixed number of complete experts.<n>We study this dependency by decomposing sparsely upcycled feed-forward experts into key-value channels.<n>We instantiate it in UniF-MoE, a unified framework for token-adaptive MoE computation.
  arXiv  Detail & Related papers  (2026-08-11T02:40:58Z)
• FedWeave: Rethinking the Unit of Specialization in Heterogeneous Federated MoE-LoRA [15.068543696107227]
  Federated PEFT enables LLMs to collaboratively adapt to decentralized private data without sharing raw examples.<n>Existing methods typically specialize at client granularity, implicitly assuming task-coherent clients.<n>We propose FedWeave, a framework that adopts asymmetric aggregation, separating expert aggregation from router optimization.
  arXiv  Detail & Related papers  (2026-07-29T08:47:58Z)
• Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models [50.34585122347149]
  We introduce Expert Tying, an architectural modification that shares expert parameters across consecutive transformer layers.<n>We evaluate this approach across common, state-of-the-art architectures, including OLMoE, Qwen3, and DeepSeek-style MoEs.
  arXiv  Detail & Related papers  (2026-06-15T15:08:09Z)
• FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models [15.447455377430634]
  FourierMoE is a frequency-adaptive router to dispatch tokens to experts specialized in distinct frequency bands.<n>It consistently outperforms competitive baselines in both single-task and multi-task settings.<n>Results highlight the promise of spectral-domain expert adaptation as an effective and parameter-efficient paradigm for LLM fine-tuning.
  arXiv  Detail & Related papers  (2026-04-02T08:30:06Z)
• CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging [49.87105462292961]
  Core Space Mixture of LoRA (bfCoMoL) is a novel MoE-LoRA framework that incorporates expert diversity, parameter efficiency, and fine-grained adaptation.<n>CoMoL consistently outperforms existing methods across multiple tasks.
  arXiv  Detail & Related papers  (2026-02-28T09:40:11Z)
• How Much is Too Much? Exploring LoRA Rank Trade-offs for Retaining Knowledge and Domain Robustness [0.8757823830026766]
  Large language models are increasingly adapted to downstream tasks through fine-tuning.<n>Full supervised fine-tuning (SFT) and parameter-efficient fine-tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA), are two dominant approaches.
  arXiv  Detail & Related papers  (2025-12-17T17:44:09Z)
• Mixture-of-Transformers Learn Faster: A Theoretical Study on Classification Problems [59.94955550958074]
  We study a tractable theoretical framework in which each transformer block acts as an expert governed by a continuously trained gating network.<n>We show that expert specialization reduces gradient conflicts and makes each subtask strongly convex.<n>We prove that the training drives the expected prediction loss to near zero in $O(log(epsilon-1)$ steps, significantly improving over the $O(epsilon-1)$ rate for a single transformer.
  arXiv  Detail & Related papers  (2025-10-30T21:07:36Z)
• Adaptive Shared Experts with LoRA-Based Mixture of Experts for Multi-Task Learning [49.90176890917986]
  Mixture-of-Experts (MoE) has emerged as a powerful framework for multi-task learning (MTL)<n>Existing MoE-MTL methods often rely on single-task pretrained backbones and suffer from redundant adaptation and inefficient knowledge sharing.<n>We propose adaptive shared experts (ASE) within a low-rank adaptation (LoRA) based MoE, where shared experts are assigned router-computed gating weights jointly normalized with sparse experts.
  arXiv  Detail & Related papers  (2025-10-01T06:49:19Z)
• FFT-MoE: Efficient Federated Fine-Tuning for Foundation Models via Large-scale Sparse MoE under Heterogeneous Edge [7.976167864455345]
  Federated Learning (FL) offers a compelling solution through Federated Fine-Tuning (FFT)<n>We propose FFT MoE, a novel FFT framework that replaces LoRA with sparse Mixture of Experts (MoE) adapters.<n>MoE consistently outperforms state of the art FFT baselines in generalization performance and training efficiency.
  arXiv  Detail & Related papers  (2025-08-26T04:09:18Z)
• Multi-Task Dense Prediction via Mixture of Low-Rank Experts [35.11968315125389]
  We present a novel decoder-focused method for multi-task dense prediction, called Mixture-of-Low-Rank-Experts (MLoRE) To model the global task relationships, MLoRE adds a generic convolution path to the original MoE structure, where each task feature can go through this path for explicit parameter sharing. Our experiments show that our MLoRE achieves superior performance compared to previous state-of-the-art methods on all metrics.
  arXiv  Detail & Related papers  (2024-03-26T14:40:17Z)
• Mitigating the Alignment Tax of RLHF [76.4300447532456]
  aligning LLMs under Reinforcement Learning with Human Feedback can lead to forgetting pretrained abilities, also known as the alignment tax. We propose model averaging to maximize alignment performance while incurring minimal alignment tax. We validate HMA's performance across a range of RLHF algorithms over OpenLLaMA-3B and further extend our findings to Mistral-7B.
  arXiv  Detail & Related papers  (2023-09-12T14:16:54Z)
• Deep Frequency Filtering for Domain Generalization [55.66498461438285]
  Deep Neural Networks (DNNs) have preferences for some frequency components in the learning process. We propose Deep Frequency Filtering (DFF) for learning domain-generalizable features. We show that applying our proposed DFF on a plain baseline outperforms the state-of-the-art methods on different domain generalization tasks.
  arXiv  Detail & Related papers  (2022-03-23T05:19:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.