Adaptive Federated Distillation for Multi-Domain Non-IID Textual Data

• URL: http://arxiv.org/abs/2508.20557v1
• Date: Thu, 28 Aug 2025 08:51:14 GMT
• Title: Adaptive Federated Distillation for Multi-Domain Non-IID Textual Data
• Authors: Jiahao Xiao, Jiangming Liu,
• Abstract summary: We introduce a comprehensive set of multi-domain non-IID scenarios and propose a unified benchmarking framework that includes diverse data.<n> Experimental results demonstrate that our models capture the diversity of local clients and achieve better performance compared to the existing works.
• Score: 6.819856310521865
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The widespread success of pre-trained language models has established a new training paradigm, where a global PLM is fine-tuned using task-specific data from local clients. The local data are highly different from each other and can not capture the global distribution of the whole data in real world. To address the challenges of non-IID data in real environments, privacy-preserving federated distillation has been proposed and highly investigated. However, previous experimental non-IID scenarios are primarily identified with the label (output) diversity, without considering the diversity of language domains (input) that is crucial in natural language processing. In this paper, we introduce a comprehensive set of multi-domain non-IID scenarios and propose a unified benchmarking framework that includes diverse data. The benchmark can be used to evaluate the federated learning framework in a real environment. To this end, we propose an Adaptive Federated Distillation (AdaFD) framework designed to address multi-domain non-IID challenges in both homogeneous and heterogeneous settings. Experimental results demonstrate that our models capture the diversity of local clients and achieve better performance compared to the existing works. The code for this paper is available at: https://github.com/jiahaoxiao1228/AdaFD.

Related papers

• FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning [103.45987800174724]
  Federated Learning (FL) confronts a significant challenge known as data heterogeneity, which impairs model performance and convergence.<n>We propose textbfFedGPS, a novel framework that seamlessly integrates statistical distribution and gradient information from others.
  arXiv  Detail & Related papers  (2025-10-23T06:10:11Z)
• UniVarFL: Uniformity and Variance Regularized Federated Learning for Heterogeneous Data [2.6733991338938026]
  UniVarFL is a novel FL framework that emulates IID-like training dynamics directly at the client level.<n>Experiments on multiple benchmark datasets demonstrate that UniVarFL outperforms existing methods in accuracy.
  arXiv  Detail & Related papers  (2025-06-09T19:25:35Z)
• AFCL: Analytic Federated Continual Learning for Spatio-Temporal Invariance of Non-IID Data [45.66391633579935]
  Federated Continual Learning (FCL) enables distributed clients to collaboratively train a global model from online task streams.<n>FCL methods face challenges of both spatial data heterogeneity among distributed clients and temporal data heterogeneity across online tasks.<n>We propose a gradient-free method, named Analytic Federated Continual Learning (AFCL), by deriving analytical (i.e., closed-form) solutions from frozen extracted features.
  arXiv  Detail & Related papers  (2025-05-18T05:55:09Z)
• Federated Learning with Domain Shift Eraser [16.685557256108265]
  Federated learning (FL) is emerging as a promising technique for collaborative learning without local data leaving their devices.<n>We propose a novel FL framework, Federated Domain Shift Eraser (FDSE), to improve model performance.
  arXiv  Detail & Related papers  (2025-03-17T11:10:31Z)
• STHFL: Spatio-Temporal Heterogeneous Federated Learning [39.32313754519315]
  Federated learning is a new framework that protects data privacy and allows multiple devices to cooperate in training machine learning models.<n>Previous studies have proposed multiple approaches to eliminate the challenges posed by non-iid data and inter-domain issues.<n>We propose a novel setting named textbfSpatio-temporal Heterogeneity Federated Learning (STHFL). Specially, the Global-Local Dynamic Prototype (GLDP) framework is designed for STHFL.
  arXiv  Detail & Related papers  (2025-01-10T08:15:02Z)
• Unified Language-driven Zero-shot Domain Adaptation [55.64088594551629]
  Unified Language-driven Zero-shot Domain Adaptation (ULDA) is a novel task setting. It enables a single model to adapt to diverse target domains without explicit domain-ID knowledge.
  arXiv  Detail & Related papers  (2024-04-10T16:44:11Z)
• Dual-Personalizing Adapter for Federated Foundation Models [35.863585349109385]
  We propose a Federated Dual-Personalizing Adapter architecture to handle test-time distribution shifts simultaneously.<n>The effectiveness of the proposed method has been evaluated on benchmark datasets across different NLP tasks.
  arXiv  Detail & Related papers  (2024-03-28T08:19:33Z)
• Generalizable Metric Network for Cross-domain Person Re-identification [55.71632958027289]
  Cross-domain (i.e., domain generalization) scene presents a challenge in Re-ID tasks. Most existing methods aim to learn domain-invariant or robust features for all domains. We propose a Generalizable Metric Network (GMN) to explore sample similarity in the sample-pair space.
  arXiv  Detail & Related papers  (2023-06-21T03:05:25Z)
• FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction [48.85303253333453]
  Federated learning (FL) allows multiple clients to collectively train a high-performance global model without sharing their private data. We propose a novel federated learning algorithm with local drift decoupling and correction (FedDC) Our FedDC only introduces lightweight modifications in the local training phase, in which each client utilizes an auxiliary local drift variable to track the gap between the local model parameter and the global model parameters. Experiment results and analysis demonstrate that FedDC yields expediting convergence and better performance on various image classification tasks.
  arXiv  Detail & Related papers  (2022-03-22T14:06:26Z)
• Federated and Generalized Person Re-identification through Domain and Feature Hallucinating [88.77196261300699]
  We study the problem of federated domain generalization (FedDG) for person re-identification (re-ID) We propose a novel method, called "Domain and Feature Hallucinating (DFH)", to produce diverse features for learning generalized local and global models. Our method achieves the state-of-the-art performance for FedDG on four large-scale re-ID benchmarks.
  arXiv  Detail & Related papers  (2022-03-05T09:15:13Z)
• FDMT: A Benchmark Dataset for Fine-grained Domain Adaptation in Machine Translation [53.87731008029645]
  We present a real-world fine-grained domain adaptation task in machine translation (FDMT) The FDMT dataset consists of four sub-domains of information technology: autonomous vehicles, AI education, real-time networks and smart phone. We make quantitative experiments and deep analyses in this new setting, which benchmarks the fine-grained domain adaptation task.
  arXiv  Detail & Related papers  (2020-12-31T17:15:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.