Continual Learning with Elastic Regularization and Synthetic Replay for Federated MLLM Fine-Tuning

• URL: http://arxiv.org/abs/2607.12112v1
• Date: Mon, 13 Jul 2026 19:40:30 GMT
• Title: Continual Learning with Elastic Regularization and Synthetic Replay for Federated MLLM Fine-Tuning
• Abstract summary: Federated Continual Multimodal Learning (FedCMM) is a framework that embeds continual-learning safeguards into the federated optimization loop.<n>FedCMM consistently outperforms recent baselines on accuracy and backward transfer.
• Score: 15.445619998280243
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated fine-tuning of Multimodal Large Language Models (MLLMs) across distributed networks enables privacy-sensitive adaptation to evolving data streams, yet a fundamental obstacle prevents robust deployment in dynamic environments: catastrophic forgetting, wherein sequential task updates erase previously acquired knowledge across visual, linguistic, and cross-modal representations. Addressing this challenge is especially critical for autonomous networked AI operating in safety-sensitive domains, such as content moderation, where reliable retention of prior knowledge underpins system integrity. To overcome this, we propose Federated Continual Multimodal Learning (FedCMM), a framework that embeds continual-learning safeguards into the federated optimization loop at three complementary levels. At the parameter level, modality-aware elastic weight consolidation computes separate Fisher information matrices for the vision encoder, language backbone, and cross-modal projector, providing granular, asymmetry-aware protection against modality-specific forgetting. At the data level, each client trains a lightweight local generative replay module to synthesize raw-data-free embedding-level multimodal replay tuples without any raw data sharing. At the aggregation level, Task-similarity-aware gradient aggregation autonomously filters and reweights client updates by gradient cosine similarity, suppressing conflicting directions and stabilizing the global learning trajectory. Extensive experiments on two benchmarks demonstrate that FedCMM consistently outperforms recent baselines on accuracy and backward transfer, confirming that holistic, modality-aware optimization enables robust evolutive adaptation across heterogeneous networked AI deployments.

Related papers

• Diffusion Language Models for Mobile Edge Agentic AI: Foundations, Applications, and Challenges [58.887152635127165]
  Diffusion language models (DLMs) offer a non-autoregressive alternative for mobile edge agentic artificial intelligence (AI)<n>DLMs can update multiple uncertain tokens in parallel and exploit bidirectional context throughout the generation process.<n>This survey reviews DLM foundations and analyzes their suitability for edge settings under latency, memory, energy, bandwidth, privacy, and reliability constraints.
  arXiv  Detail & Related papers  (2026-09-04T06:16:49Z)
• LargeMonitor: Monitoring Online Task-Free Continual Learning via Large Pretrained Models [5.257217773590746]
  Online task-free continual learning (TFCL) requires intelligent agents to sequentially accumulate knowledge from an unbounded, non-stationary data stream.<n>Existing online TFCL paradigms rely on parameter-efficient prompt tuning or dynamic structure expansion driven by training-coupled optimization dynamics.<n>We propose LargeMonitor, a framework that leverages large pretrained foundation models to autonomously orchestrate task-free continuous adaptation.
  arXiv  Detail & Related papers  (2026-06-08T12:41:15Z)
• AlignFed: Alignment-Aware Asynchronous Federated Fine-Tuning for Large Language Models in Heterogeneous Edge Environments [7.335282879368365]
  Large Language Models (LLMs) have been widely deployed across various scenarios, including autonomous driving, industrial inspection, and personalized IoT services.<n>This paper proposes AlignFed, an asynchronous federated fine-tuning framework for LLMs tailored to heterogeneous edge environments.
  arXiv  Detail & Related papers  (2026-06-06T14:28:36Z)
• FLARE: Diffusion for Hybrid Language Model [72.60770374799634]
  FLARE is a systematic conversion framework for hybrid-attention large language models.<n>It combines a token-equal AR-and-diffusion objective, hardware-aware kernels, and unified inference, enabling one checkpoint to support both AR-style verified decoding and diffusion-style parallel denoising.<n>Our results suggest practical dLLMs are limited not only by decoding algorithms, but also by transfer data quality and the training inefficiency of current block-diffusion objectives.
  arXiv  Detail & Related papers  (2026-06-01T06:58:15Z)
• Revitalizing the Beginning: Avoiding Storage Dependency for Model Merging in Continual Learning [57.10440766103372]
  Trajectory Regularized Merging (TRM) is a framework that reformulates the merging phase as an optimization process within an augmented trajectory subspace.<n>Our framework integrates three synergistic objectives including task alignment, prediction consistency, and gradient responsiveness to concurrently preserve merged model's historical stability and re-activate optimization dynamics.
  arXiv  Detail & Related papers  (2026-05-08T14:07:32Z)
• SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utility [71.76390626651254]
  We propose SPARD, a framework that establishes an automated, self-paced curriculum by perceiving learning progress to adjust dynamically multi-objective reward weights and data importance.<n>Extensive experiments across multiple benchmarks demonstrate that SPARD significantly enhances model capabilities across all domains.
  arXiv  Detail & Related papers  (2026-04-09T05:37:22Z)
• Adaptive Dual-Weighting Framework for Federated Learning via Out-of-Distribution Detection [53.45696787935487]
  Federated Learning (FL) enables collaborative model training across large-scale distributed service nodes.<n>In real-world service-oriented deployments, data generated by heterogeneous users, devices, and application scenarios are inherently non-IID.<n>We propose FLood, a novel FL framework inspired by out-of-distribution (OOD) detection.
  arXiv  Detail & Related papers  (2026-02-01T05:54:59Z)
• Beyond Static Datasets: Robust Offline Policy Optimization via Vetted Synthetic Transitions [4.359780028396042]
  We present MoReBRAC, a model-based framework that addresses the distributional shift between the static dataset and the learned policy.<n>We implement a hierarchical uncertainty pipeline integrating Variational Autoencoder (VAE) manifold detection, model sensitivity analysis, and Monte Carlo (MC) dropout.<n>Our results on D4RL Gym-MuJoCo benchmarks reveal significant performance gains, particularly in random'' and suboptimal'' data regimes.
  arXiv  Detail & Related papers  (2026-01-26T03:38:27Z)
• SMoFi: Step-wise Momentum Fusion for Split Federated Learning on Heterogeneous Data [11.41105795202393]
  Split Federated Learning uses rich computing resources at a central server to train model partitions.<n>Data heterogeneity across silos presents a major challenge undermining the convergence speed and accuracy of the global model.<n>This paper introduces Step-wise Momentum Fusion (SMoFi), an effective and lightweight framework that counteracts gradient divergence.
  arXiv  Detail & Related papers  (2025-11-13T00:21:05Z)
• Unsupervised Online 3D Instance Segmentation with Synthetic Sequences and Dynamic Loss [52.28880405119483]
  Unsupervised online 3D instance segmentation is a fundamental yet challenging task.<n>Existing methods, such as UNIT, have made progress in this direction but remain constrained by limited training diversity.<n>We propose a new framework that enriches the training distribution through synthetic point cloud sequence generation.
  arXiv  Detail & Related papers  (2025-09-27T08:53:27Z)
• FedAWA: Adaptive Optimization of Aggregation Weights in Federated Learning Using Client Vectors [50.131271229165165]
  Federated Learning (FL) has emerged as a promising framework for distributed machine learning.<n>Data heterogeneity resulting from differences across user behaviors, preferences, and device characteristics poses a significant challenge for federated learning.<n>We propose Adaptive Weight Aggregation (FedAWA), a novel method that adaptively adjusts aggregation weights based on client vectors during the learning process.
  arXiv  Detail & Related papers  (2025-03-20T04:49:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.