CoAst: Validation-Free Contribution Assessment for Federated Learning based on Cross-Round Valuation

• URL: http://arxiv.org/abs/2409.02495v1
• Date: Wed, 4 Sep 2024 07:46:28 GMT
• Title: CoAst: Validation-Free Contribution Assessment for Federated Learning based on Cross-Round Valuation
• Authors: Hao Wu, Likun Zhang, Shucheng Li, Fengyuan Xu, Sheng Zhong,
• Abstract summary: CoAst is a practical method to assess the contribution without access to any validation data. CoAst has comparable assessment reliability to existing validation-based methods and outperforms existing validation-free methods.
• Score: 10.579048525756797
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the federated learning (FL) process, since the data held by each participant is different, it is necessary to figure out which participant has a higher contribution to the model performance. Effective contribution assessment can help motivate data owners to participate in the FL training. Research works in this field can be divided into two directions based on whether a validation dataset is required. Validation-based methods need to use representative validation data to measure the model accuracy, which is difficult to obtain in practical FL scenarios. Existing validation-free methods assess the contribution based on the parameters and gradients of local models and the global model in a single training round, which is easily compromised by the stochasticity of model training. In this work, we propose CoAst, a practical method to assess the FL participants' contribution without access to any validation data. The core idea of CoAst involves two aspects: one is to only count the most important part of model parameters through a weights quantization, and the other is a cross-round valuation based on the similarity between the current local parameters and the global parameter updates in several subsequent communication rounds. Extensive experiments show that CoAst has comparable assessment reliability to existing validation-based methods and outperforms existing validation-free methods.

Related papers

• Redefining Contributions: Shapley-Driven Federated Learning [3.9539878659683363]
  Federated learning (FL) has emerged as a pivotal approach in machine learning. It is challenging to ensure global model convergence when participants do not contribute equally and/or honestly. This paper proposes a novel contribution assessment method called ShapFed for fine-grained evaluation of participant contributions in FL.
  arXiv  Detail & Related papers  (2024-06-01T22:40:31Z)
• Data vs. Model Machine Learning Fairness Testing: An Empirical Study [23.535630175567146]
  We take the first steps towards evaluating a more holistic approach by testing for fairness both before and after model training. We evaluate the effectiveness of the proposed approach using an empirical analysis of the relationship between model dependent and independent fairness metrics. Our results indicate that testing for fairness prior to training can be a cheap'' and effective means of catching a biased data collection process early.
  arXiv  Detail & Related papers  (2024-01-15T14:14:16Z)
• TEA: Test-time Energy Adaptation [67.4574269851666]
  Test-time adaptation (TTA) aims to improve model generalizability when test data diverges from training distribution. We propose a novel energy-based perspective, enhancing the model's perception of target data distributions.
  arXiv  Detail & Related papers  (2023-11-24T10:49:49Z)
• Data Valuation and Detections in Federated Learning [4.899818550820576]
  Federated Learning (FL) enables collaborative model training while preserving the privacy of raw data. A challenge in this framework is the fair and efficient valuation of data, which is crucial for incentivizing clients to contribute high-quality data in the FL task. This paper introduces a novel privacy-preserving method for evaluating client contributions and selecting relevant datasets without a pre-specified training algorithm in an FL task.
  arXiv  Detail & Related papers  (2023-11-09T12:01:32Z)
• Integrating Local Real Data with Global Gradient Prototypes for Classifier Re-Balancing in Federated Long-Tailed Learning [60.41501515192088]
  Federated Learning (FL) has become a popular distributed learning paradigm that involves multiple clients training a global model collaboratively. The data samples usually follow a long-tailed distribution in the real world, and FL on the decentralized and long-tailed data yields a poorly-behaved global model. In this work, we integrate the local real data with the global gradient prototypes to form the local balanced datasets.
  arXiv  Detail & Related papers  (2023-01-25T03:18:10Z)
• Provable Fairness for Neural Network Models using Formal Verification [10.90121002896312]
  We propose techniques to emphprove fairness using recently developed formal methods that verify properties of neural network models. We show that through proper training, we can reduce unfairness by an average of 65.4% at a cost of less than 1% in AUC score.
  arXiv  Detail & Related papers  (2022-12-16T16:54:37Z)
• Exploring validation metrics for offline model-based optimisation with diffusion models [50.404829846182764]
  In model-based optimisation (MBO) we are interested in using machine learning to design candidates that maximise some measure of reward with respect to a black box function called the (ground truth) oracle. While an approximation to the ground oracle can be trained and used in place of it during model validation to measure the mean reward over generated candidates, the evaluation is approximate and vulnerable to adversarial examples. This is encapsulated under our proposed evaluation framework which is also designed to measure extrapolation.
  arXiv  Detail & Related papers  (2022-11-19T16:57:37Z)
• FairIF: Boosting Fairness in Deep Learning via Influence Functions with Validation Set Sensitive Attributes [51.02407217197623]
  We propose a two-stage training algorithm named FAIRIF. It minimizes the loss over the reweighted data set where the sample weights are computed. We show that FAIRIF yields models with better fairness-utility trade-offs against various types of bias.
  arXiv  Detail & Related papers  (2022-01-15T05:14:48Z)
• Scalable Personalised Item Ranking through Parametric Density Estimation [53.44830012414444]
  Learning from implicit feedback is challenging because of the difficult nature of the one-class problem. Most conventional methods use a pairwise ranking approach and negative samplers to cope with the one-class problem. We propose a learning-to-rank approach, which achieves convergence speed comparable to the pointwise counterpart.
  arXiv  Detail & Related papers  (2021-05-11T03:38:16Z)
• Towards Model-Agnostic Post-Hoc Adjustment for Balancing Ranking Fairness and Algorithm Utility [54.179859639868646]
  Bipartite ranking aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data. There have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups. We propose a model post-processing framework for balancing them in the bipartite ranking scenario.
  arXiv  Detail & Related papers  (2020-06-15T10:08:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.