Fisher-Routed Mixture of Experts for Federated Class-Incremental Learning

• URL: http://arxiv.org/abs/2606.28835v1
• Date: Sat, 27 Jun 2026 09:45:52 GMT
• Title: Fisher-Routed Mixture of Experts for Federated Class-Incremental Learning
• Abstract summary: Federated Learning (FL) emerged as a promising distributed machine learning paradigm.<n> extending FL to the class incremental learning scenarios introduces unique challenges.<n>We propose textbfFisher-Routed textbfMitextbfXture of Experts for textbfFederated Class-Incremental Learning (textscFedFMX)
• Score: 4.9788177236136635
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated Learning (FL) emerged as a promising distributed machine learning paradigm. However, extending FL to the class incremental learning scenarios introduces unique challenges: 1) Capacity conflict and catastrophic forgetting from the shared model overloading, 2) Heterogeneity from Non-Independent and Identically Distributed (Non-IID) data, and 3) Synchronized class misalignment. In this paper, we propose \textbf{F}isher-Routed \textbf{M}i\textbf{X}ture of Experts for \textbf{Fed}erated Class-Incremental Learning (\textsc{FedFMX}), a novel framework to address these challenges via adaptive expert specialization across clients. The crucial insight is to route each sample to an expert subset that jointly optimizes knowledge acquisition and retention. Specifically, we introduce a Fisher-Routed Expert Scoring (FRES) module to estimate expert importance via Fisher-based stability cost and gradient-based plasticity gain. Then, we design an Adaptive Expert Selection (AES) module by quantifying marginal contributions for adaptive expert subset determination. Finally, by the routing-aware regularization (RAR), we achieve load balance and efficient FL training. We theoretically prove the $\mathcal{O}(T^{-1})$ convergence rate. Extensive experiments on multiple benchmarks compared with state-of-the-art methods demonstrate the superiority of \textsc{FedFMX}.

Related papers

• VICAL: Vicinal Consistency Alignment for Long-Tailed Visual Recognition [32.48884030865545]
  VICAL is a framework that improves long-tailed recognition not by enforcing expert diversity, but by reducing prediction variance.<n>Our approach comprises two key components: Self-Consistency Learning and Deep Ensemble Distillation.<n>Our experiments on CIFAR-LT, ImageNet-LT, and iist 2018 show that VICAL consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-09-04T09:55:12Z)
• Mastering the Minority: An Uncertainty-guided Multi-Expert Framework for Challenging-tailed Sequence Learning [34.71004757894417]
  We propose the Uncertainty-based Multi-Expert fusion network (UME) framework to Master the Minority classes effectively.<n>UME is designed with three core innovations: First, we employ Ensemble LoRA for parameter-efficient modeling, significantly reducing the trainable parameter count.<n>We achieve a performance gain of up to 17.97% over the best baseline on individual categories, while reducing trainable parameters by up to 10.32%.
  arXiv  Detail & Related papers  (2026-03-16T13:34:56Z)
• How to Set the Learning Rate for Large-Scale Pre-training? [73.03133634525635]
  We formalize this investigation into two distinct research paradigms: Fitting and Transfer.<n>Within the Fitting Paradigm, we introduce a Scaling Law for search factor, effectively reducing the search complexity from O(n3) to O(n*C_D*C_) via predictive modeling.<n>We extend the principles of $$Transfer to the Mixture of Experts (MoE) architecture, broadening its applicability to encompass model depth, weight decay, and token horizons.
  arXiv  Detail & Related papers  (2026-01-08T15:55:13Z)
• FLEX-MoE: Federated Mixture-of-Experts with Load-balanced Expert Assignment [38.27527504479237]
  Mixture-of-Experts (MoE) models enable scalable neural networks through conditional computation.<n>Our approach introduces client-expert fitness scores that quantify the expert suitability for local datasets through training feedback.<n>Our comprehensive experiments on three different datasets demonstrate the superior performance of the proposed FLEX-MoE.
  arXiv  Detail & Related papers  (2025-12-28T20:32:13Z)
• Supervised Fine-Tuning or Contrastive Learning? Towards Better Multimodal LLM Reranking [56.46309219272326]
  For large language models (LLMs), classification via supervised fine-tuning (SFT) predicts ''yes'' (resp. ''no'') token for relevant (resp. irrelevant) pairs.<n>This divergence raises a central question: which objective is intrinsically better suited to LLM-based reranking, and what mechanism underlies the difference?<n>We conduct a comprehensive comparison and analysis between CL and SFT for reranking, taking the universal multimodal retrieval (UMR) as the experimental playground.
  arXiv  Detail & Related papers  (2025-10-16T16:02:27Z)
• MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources [113.33902847941941]
  Variance-Aware Sampling (VAS) is a data selection strategy guided by Variance Promotion Score (VPS)<n>We release large-scale, carefully curated resources containing 1.6M long CoT cold-start data and 15k RL QA pairs.<n> Experiments across mathematical reasoning benchmarks demonstrate the effectiveness of both the curated data and the proposed VAS.
  arXiv  Detail & Related papers  (2025-09-25T14:58:29Z)
• Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectories [58.988535279557546]
  We introduce textbf sycophancy Mitigation through Adaptive Reasoning Trajectories.<n>We show that SMART significantly reduces sycophantic behavior while preserving strong performance on out-of-distribution inputs.
  arXiv  Detail & Related papers  (2025-09-20T17:09:14Z)
• FedPref: Federated Learning Across Heterogeneous Multi-objective Preferences [2.519319150166215]
  Federated Learning (FL) is a distributed machine learning strategy developed for settings where training data is owned by distributed devices and cannot be shared.<n>The application of FL to real-world settings brings additional challenges associated with heterogeneity between participants.<n>We propose FedPref, a first algorithm designed to facilitate personalised FL in this setting.
  arXiv  Detail & Related papers  (2025-01-23T12:12:59Z)
• Client-Centric Federated Adaptive Optimization [78.30827455292827]
  Federated Learning (FL) is a distributed learning paradigm where clients collaboratively train a model while keeping their own data private.<n>We propose Federated-Centric Adaptive Optimization, which is a class of novel federated optimization approaches.
  arXiv  Detail & Related papers  (2025-01-17T04:00:50Z)
• Over-the-Air Fair Federated Learning via Multi-Objective Optimization [52.295563400314094]
  We propose an over-the-air fair federated learning algorithm (OTA-FFL) to train fair FL models.<n>Experiments demonstrate the superiority of OTA-FFL in achieving fairness and robust performance.
  arXiv  Detail & Related papers  (2025-01-06T21:16:51Z)
• Learning Mixtures of Experts with EM: A Mirror Descent Perspective [28.48469221248906]
  Classical Mixtures of Experts (MoE) are Machine Learning models that involve the input space, with a separate "expert" model trained on each partition.<n>We study theoretical guarantees of the Expectation Maximization (EM) algorithm for the training of MoE models.
  arXiv  Detail & Related papers  (2024-11-09T03:44:09Z)
• FedEGG: Federated Learning with Explicit Global Guidance [90.04705121816185]
  Federated Learning (FL) holds great potential for diverse applications owing to its privacy-preserving nature.<n>Existing methods help address these challenges via optimization-based client constraints, adaptive client selection, or the use of pre-trained models or synthetic data.<n>We present bftextFedEGG, a new FL algorithm that constructs a global guiding task using a well-defined, easy-to-converge learning task.
  arXiv  Detail & Related papers  (2024-04-18T04:25:21Z)
• Disentangled Federated Learning for Tackling Attributes Skew via Invariant Aggregation and Diversity Transferring [104.19414150171472]
  Attributes skews the current federated learning (FL) frameworks from consistent optimization directions among the clients. We propose disentangled federated learning (DFL) to disentangle the domain-specific and cross-invariant attributes into two complementary branches. Experiments verify that DFL facilitates FL with higher performance, better interpretability, and faster convergence rate, compared with SOTA FL methods.
  arXiv  Detail & Related papers  (2022-06-14T13:12:12Z)
• Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customization [107.72786199113183]
  Federated learning (FL) provides a distributed learning framework for multiple participants to collaborate learning without sharing raw data. In this paper, we propose a novel Split-Mix FL strategy for heterogeneous participants that, once training is done, provides in-situ customization of model sizes and robustness.
  arXiv  Detail & Related papers  (2022-03-18T04:58:34Z)
• On the Convergence of Heterogeneous Federated Learning with Arbitrary Adaptive Online Model Pruning [15.300983585090794]
  We present a unifying framework for heterogeneous FL algorithms with em arbitrary adaptive online model pruning. In particular, we prove that under certain sufficient conditions, these algorithms converge to a stationary point of standard FL for general smooth cost functions. We illuminate two key factors impacting convergence: pruning-induced noise and minimum coverage index.
  arXiv  Detail & Related papers  (2022-01-27T20:43:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.