FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout

• URL: http://arxiv.org/abs/2102.13451v2
• Date: Mon, 1 Mar 2021 09:16:03 GMT
• Title: FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
• Authors: Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos I. Venieris and Nicholas D. Lane
• Abstract summary: We introduce Ordered Dropout, a mechanism that achieves an ordered, nested representation of knowledge in Neural Networks. We employ this technique, along with a self-distillation methodology, in the realm of Federated Learning in a framework called FjORD. FjORD consistently leads to significant performance gains over state-of-the-art baselines, while maintaining its nested structure.
• Score: 16.250862114257277
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated Learning (FL) has been gaining significant traction across different ML tasks, ranging from vision to keyboard predictions. In large-scale deployments, client heterogeneity is a fact, and constitutes a primary problem for fairness, training performance and accuracy. Although significant efforts have been made into tackling statistical data heterogeneity, the diversity in the processing capabilities and network bandwidth of clients, termed as system heterogeneity, has remained largely unexplored. Current solutions either disregard a large portion of available devices or set a uniform limit on the model's capacity, restricted by the least capable participants. In this work, we introduce Ordered Dropout, a mechanism that achieves an ordered, nested representation of knowledge in Neural Networks and enables the extraction of lower footprint submodels without the need of retraining. We further show that for linear maps our Ordered Dropout is equivalent to SVD. We employ this technique, along with a self-distillation methodology, in the realm of FL in a framework called FjORD. FjORD alleviates the problem of client system heterogeneity by tailoring the model width to the client's capabilities. Extensive evaluation on both CNNs and RNNs across diverse modalities shows that FjORD consistently leads to significant performance gains over state-of-the-art baselines, while maintaining its nested structure.

Related papers

• Adaptive Dual-Weighting Framework for Federated Learning via Out-of-Distribution Detection [53.45696787935487]
  Federated Learning (FL) enables collaborative model training across large-scale distributed service nodes.<n>In real-world service-oriented deployments, data generated by heterogeneous users, devices, and application scenarios are inherently non-IID.<n>We propose FLood, a novel FL framework inspired by out-of-distribution (OOD) detection.
  arXiv  Detail & Related papers  (2026-02-01T05:54:59Z)
• Rethinking Federated Graph Foundation Models: A Graph-Language Alignment-based Approach [8.517604507672262]
  Recent studies of federated graph foundational models (FedGFMs) break the idealized and untenable assumption of having centralized data storage to train graph foundation models.<n>Existing studies that project aligned generalizable knowledge onto a discrete token space via vector-quantized backbones suffer from irreversible knowledge loss during the quantization process.
  arXiv  Detail & Related papers  (2026-01-29T07:50:00Z)
• FairViT-GAN: A Hybrid Vision Transformer with Adversarial Debiasing for Fair and Explainable Facial Beauty Prediction [0.0]
  We propose textbfFairViT-GAN, a novel hybrid framework for facial beauty prediction.<n>We show that FairViT-GAN sets a new state-of-the-art in predictive accuracy, achieving a Pearson Correlation of textbf0.9230 and reducing RMSE to textbf0.2650.<n>Our analysis reveals a remarkable textbf82.9% reduction in the performance gap between ethnic subgroups, with the adversary's classification accuracy dropping to near-random chance (52.1%)
  arXiv  Detail & Related papers  (2025-09-28T12:55:31Z)
• Deep Hierarchical Learning with Nested Subspace Networks [53.71337604556311]
  We propose Nested Subspace Networks (NSNs) for large neural networks.<n>NSNs enable a single model to be dynamically and granularly adjusted across a continuous spectrum of compute budgets.<n>We show that NSNs can be surgically applied to pre-trained LLMs and unlock a smooth and predictable compute-performance frontier.
  arXiv  Detail & Related papers  (2025-09-22T15:13:14Z)
• Widening the Network Mitigates the Impact of Data Heterogeneity on FedAvg [6.185573921868495]
  Federated learning (FL) enables decentralized clients to train a model collaboratively without sharing local data.<n>We prove that the impact of data heterogeneity diminishes as the width of neural networks increases, ultimately vanishing when the width approaches infinity.<n>In the infinite-width regime, we further prove that both the global and local models in FedAvg behave as linear models, and that FedAvg achieves the same generalization performance as centralized learning with the same number of GD iterations.
  arXiv  Detail & Related papers  (2025-08-18T02:22:55Z)
• FedSKD: Aggregation-free Model-heterogeneous Federated Learning using Multi-dimensional Similarity Knowledge Distillation [7.944298319589845]
  Federated learning (FL) enables privacy-preserving collaborative model training without direct data sharing. Model-heterogeneous FL (MHFL) allows clients to train personalized models with heterogeneous architectures tailored to their computational resources and application-specific needs. While peer-to-peer (P2P) FL removes server dependence, it suffers from model drift and knowledge dilution, limiting its effectiveness in heterogeneous settings. We propose FedSKD, a novel MHFL framework that facilitates direct knowledge exchange through round-robin model circulation.
  arXiv  Detail & Related papers  (2025-03-23T05:33:10Z)
• FedPref: Federated Learning Across Heterogeneous Multi-objective Preferences [2.519319150166215]
  Federated Learning (FL) is a distributed machine learning strategy developed for settings where training data is owned by distributed devices and cannot be shared. The application of FL to real-world settings brings additional challenges associated with heterogeneity between participants. We propose FedPref, a first algorithm designed to facilitate personalised FL in this setting.
  arXiv  Detail & Related papers  (2025-01-23T12:12:59Z)
• Client-Centric Federated Adaptive Optimization [78.30827455292827]
  Federated Learning (FL) is a distributed learning paradigm where clients collaboratively train a model while keeping their own data private. We propose Federated-Centric Adaptive Optimization, which is a class of novel federated optimization approaches.
  arXiv  Detail & Related papers  (2025-01-17T04:00:50Z)
• Over-the-Air Fair Federated Learning via Multi-Objective Optimization [52.295563400314094]
  We propose an over-the-air fair federated learning algorithm (OTA-FFL) to train fair FL models. Experiments demonstrate the superiority of OTA-FFL in achieving fairness and robust performance.
  arXiv  Detail & Related papers  (2025-01-06T21:16:51Z)
• FedMAC: Tackling Partial-Modality Missing in Federated Learning with Cross-Modal Aggregation and Contrastive Regularization [18.276480518649404]
  Federated Learning (FL) is a method for training machine learning models using distributed data sources.<n>This study proposes a novel framework named FedMAC, designed to address multi-modality missing under conditions of partial-modality missing in FL.
  arXiv  Detail & Related papers  (2024-10-04T01:24:02Z)
• FedHPL: Efficient Heterogeneous Federated Learning with Prompt Tuning and Logit Distillation [32.305134875959226]
  Federated learning (FL) is a privacy-preserving paradigm that enables distributed clients to collaboratively train models with a central server. We propose FedHPL, a parameter-efficient unified $textbfFed$erated learning framework for $textbfH$eterogeneous settings. We show that our framework outperforms state-of-the-art FL approaches, with less overhead and training rounds.
  arXiv  Detail & Related papers  (2024-05-27T15:25:32Z)
• Fed-CO2: Cooperation of Online and Offline Models for Severe Data Heterogeneity in Federated Learning [14.914477928398133]
  Federated Learning (FL) has emerged as a promising distributed learning paradigm. The effectiveness of FL is highly dependent on the quality of the data that is being used for training. We propose Fed-CO$_2$, a universal FL framework that handles both label distribution skew and feature skew.
  arXiv  Detail & Related papers  (2023-12-21T15:12:12Z)
• Fed-QSSL: A Framework for Personalized Federated Learning under Bitwidth and Data Heterogeneity [14.313847382199059]
  Federated quantization-based self-supervised learning scheme (Fed-QSSL) designed to address heterogeneity in FL systems. Fed-QSSL deploys de-quantization, weighted aggregation and re-quantization, ultimately creating models personalized to both data distribution and specific infrastructure of each client's device.
  arXiv  Detail & Related papers  (2023-12-20T19:11:19Z)
• One-Shot Federated Learning with Classifier-Guided Diffusion Models [44.604485649167216]
  One-shot federated learning (OSFL) has gained attention in recent years due to its low communication cost. In this paper, we explore the novel opportunities that diffusion models bring to OSFL and propose FedCADO. FedCADO generates data that complies with clients' distributions and subsequently training the aggregated model on the server.
  arXiv  Detail & Related papers  (2023-11-15T11:11:25Z)
• Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
  Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
  arXiv  Detail & Related papers  (2023-08-11T09:58:47Z)
• Unlocking the Potential of Federated Learning for Deeper Models [24.875271131226707]
  Federated learning (FL) is a new paradigm for distributed machine learning that allows a global model to be trained across multiple clients. We propose several technical guidelines based on reducing divergence, such as using wider models and reducing the receptive field. These approaches can greatly improve the accuracy of FL on deeper models.
  arXiv  Detail & Related papers  (2023-06-05T08:45:44Z)
• FedIN: Federated Intermediate Layers Learning for Model Heterogeneity [7.781409257429762]
  Federated learning (FL) facilitates edge devices to cooperatively train a global shared model while maintaining the training data locally and privately. In this study, we propose an FL method called Federated Intermediate Layers Learning (FedIN), supporting heterogeneous models without relying on any public dataset. Experiment results demonstrate the superior performance of FedIN in heterogeneous model environments compared to state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2023-04-03T07:20:43Z)
• Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning [86.59588262014456]
  Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint. We propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG) Our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.
  arXiv  Detail & Related papers  (2022-03-17T11:18:17Z)
• Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning [96.75889543560497]
  In many real-world problems, collecting a large number of labeled samples is infeasible. Few-shot learning is the dominant approach to address this issue, where the objective is to quickly adapt to novel categories in presence of a limited number of samples. We propose a novel training mechanism that simultaneously enforces equivariance and invariance to a general set of geometric transformations.
  arXiv  Detail & Related papers  (2021-03-01T21:14:33Z)
• Feature Quantization Improves GAN Training [126.02828112121874]
  Feature Quantization (FQ) for the discriminator embeds both true and fake data samples into a shared discrete space. Our method can be easily plugged into existing GAN models, with little computational overhead in training.
  arXiv  Detail & Related papers  (2020-04-05T04:06:50Z)
• Diversity inducing Information Bottleneck in Model Ensembles [73.80615604822435]
  In this paper, we target the problem of generating effective ensembles of neural networks by encouraging diversity in prediction. We explicitly optimize a diversity inducing adversarial loss for learning latent variables and thereby obtain diversity in the output predictions necessary for modeling multi-modal data. Compared to the most competitive baselines, we show significant improvements in classification accuracy, under a shift in the data distribution.
  arXiv  Detail & Related papers  (2020-03-10T03:10:41Z)
• When Relation Networks meet GANs: Relation GANs with Triplet Loss [110.7572918636599]
  Training stability is still a lingering concern of generative adversarial networks (GANs) In this paper, we explore a relation network architecture for the discriminator and design a triplet loss which performs better generalization and stability. Experiments on benchmark datasets show that the proposed relation discriminator and new loss can provide significant improvement on variable vision tasks.
  arXiv  Detail & Related papers  (2020-02-24T11:35:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.