Beyond performance-wise Contribution Evaluation in Federated Learning

• URL: http://arxiv.org/abs/2602.22470v1
• Date: Wed, 25 Feb 2026 23:10:13 GMT
• Title: Beyond performance-wise Contribution Evaluation in Federated Learning
• Authors: Balazs Pejo,
• Abstract summary: Federated learning offers a privacy-friendly collaborative learning framework.<n>Its success hinges on the contributions of its participants.<n>This work investigates the issue of client contributions towards a model's trustworthiness.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated learning offers a privacy-friendly collaborative learning framework, yet its success, like any joint venture, hinges on the contributions of its participants. Existing client evaluation methods predominantly focus on model performance, such as accuracy or loss, which represents only one dimension of a machine learning model's overall utility. In contrast, this work investigates the critical, yet overlooked, issue of client contributions towards a model's trustworthiness -- specifically, its reliability (tolerance to noisy data), resilience (resistance to adversarial examples), and fairness (measured via demographic parity). To quantify these multifaceted contributions, we employ the state-of-the-art approximation of the Shapley value, a principled method for value attribution. Our results reveal that no single client excels across all dimensions, which are largely independent from each other, highlighting a critical flaw in current evaluation scheme: no single metric is adequate for comprehensive evaluation and equitable rewarding allocation.

Related papers

• Private and Robust Contribution Evaluation in Federated Learning [44.1965210509968]
  Cross-silo federated learning allows multiple organizations to collaboratively train machine learning models without sharing raw data.<n>Secure aggregation protects privacy by hiding individual updates, yet it complicates contribution evaluation.<n>We introduce two marginal-difference contribution scores compatible with secure aggregation.
  arXiv  Detail & Related papers  (2026-02-25T09:27:40Z)
• Fair Feature Importance Scores via Feature Occlusion and Permutation [41.73851747821022]
  We propose two model-agnostic approaches to measure fair feature importance.<n>First, we compare model fairness before and after permuting feature values.<n>Second, we evaluate the fairness of models trained with and without a given feature.
  arXiv  Detail & Related papers  (2026-02-09T21:02:52Z)
• Reference-Specific Unlearning Metrics Can Hide the Truth: A Reality Check [60.77691669644931]
  We propose Functional Alignment for Distributional Equivalence (FADE), a novel metric that measures distributional similarity between unlearned and reference models.<n>We show that FADE captures functional alignment across the entire output distribution, providing a principled assessment of genuine unlearning.<n>These findings expose fundamental gaps in current evaluation practices and demonstrate that FADE provides a more robust foundation for developing and assessing truly effective unlearning methods.
  arXiv  Detail & Related papers  (2025-10-14T20:50:30Z)
• On the Fragility of Contribution Score Computation in Federated Learning [0.0]
  We argue that contribution scores are susceptible to significant distortions from two fundamental perspectives.<n>First, we explore how different model aggregation methods impact these scores.<n>Second, we explore vulnerabilities posed by poisoning attacks.
  arXiv  Detail & Related papers  (2025-09-24T09:22:04Z)
• DPVS-Shapley:Faster and Universal Contribution Evaluation Component in Federated Learning [1.740992908651449]
  We introduce a component called Dynamic Pruning Validation Set Shapley (DPVS-Shapley) This method accelerates the contribution assessment process by dynamically pruning the original dataset without compromising the evaluation's accuracy.
  arXiv  Detail & Related papers  (2024-10-19T13:01:44Z)
• Towards Effective Evaluations and Comparisons for LLM Unlearning Methods [97.2995389188179]
  This paper seeks to refine the evaluation of machine unlearning for large language models.<n>It addresses two key challenges -- the robustness of evaluation metrics and the trade-offs between competing goals.
  arXiv  Detail & Related papers  (2024-06-13T14:41:00Z)
• Navigating Conflicting Views: Harnessing Trust for Learning [5.776290041122041]
  We develop a computational trust-based discounting method that enhances the Evidential Multi-view framework.<n>We evaluate our method on six real-world datasets using Top-1 Accuracy, Fleiss' Kappa, and a new metric, Multi-View Agreement with Ground Truth.
  arXiv  Detail & Related papers  (2024-06-03T03:22:18Z)
• On the Volatility of Shapley-Based Contribution Metrics in Federated Learning [1.827018440608344]
  Federated learning (FL) is a collaborative and privacy-preserving Machine Learning paradigm.<n>Inaccurate allocation of contributions can undermine trust, lead to unfair compensation, and thus participants may lack the incentive to join or actively contribute to the federation.<n>We provide an extensive analysis of the discrepancies of Shapley values across a set of aggregation strategies and examine them on an overall and a per-client level.
  arXiv  Detail & Related papers  (2024-05-13T13:55:34Z)
• Toward Understanding the Influence of Individual Clients in Federated Learning [52.07734799278535]
  Federated learning allows clients to jointly train a global model without sending their private data to a central server. We defined a new notion called em-Influence, quantify this influence over parameters, and proposed an effective efficient model to estimate this metric.
  arXiv  Detail & Related papers  (2020-12-20T14:34:36Z)
• Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning [66.30909748400023]
  We propose to evaluate the summary qualities without reference summaries by unsupervised contrastive learning. Specifically, we design a new metric which covers both linguistic qualities and semantic informativeness based on BERT. Experiments on Newsroom and CNN/Daily Mail demonstrate that our new evaluation method outperforms other metrics even without reference summaries.
  arXiv  Detail & Related papers  (2020-10-05T05:04:14Z)
• Accurate and Robust Feature Importance Estimation under Distribution Shifts [49.58991359544005]
  PRoFILE is a novel feature importance estimation method. We show significant improvements over state-of-the-art approaches, both in terms of fidelity and robustness.
  arXiv  Detail & Related papers  (2020-09-30T05:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.