Trustworthy Federated Label Distribution Learning under Annotation Quality Disparity

• URL: http://arxiv.org/abs/2605.04827v2
• Date: Sun, 10 May 2026 00:57:26 GMT
• Title: Trustworthy Federated Label Distribution Learning under Annotation Quality Disparity
• Abstract summary: FedQual is a quality-aware Fed-LDL framework with two mechanisms.<n>We construct four new Fed-LDL benchmarks with controlled annotation quality disparity.
• Score: 70.60312607268871
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Label Distribution Learning (LDL) models supervision as an instance-wise probability distribution, enabling fine-grained learning under inherent ambiguity, but its success relies on high-fidelity label distributions that are costly to obtain and thus often noisy. Motivated by privacy-sensitive applications, we study Federated Label Distribution Learning (Fed-LDL), where data isolation further induces heterogeneous annotation quality across clients, making local updates unevenly reliable and breaking sample-size-based aggregation (e.g., FedAvg). To address this trust dilemma, we propose FedQual, a quality-aware Fed-LDL framework with two coupled mechanisms: (i) quality-adaptive client training guided by a global semantic anchor that calibrates low-quality clients while preserving high-quality autonomy, and (ii) reliability-aware server aggregation that reweights client contributions by effective reliable information rather than raw sample size. To enable rigorous evaluation, we construct four new Fed-LDL benchmarks (FER-LDL, FI-LDL, PIPAL-LDL, and KADID-LDL) with controlled annotation quality disparity. We further provide a theoretical guarantee showing that under heterogeneous supervision quality, client-specific calibration is strictly better than any uniform calibration. Extensive experiments on the proposed benchmarks demonstrate the effectiveness of FedQual.

Related papers

• Is Conformal Factuality for RAG-based LLMs Robust? Novel Metrics and Systematic Insights [7.594584500009769]
  Large language models (LLMs) frequently hallucinate, limiting their reliability in knowledge-intensive applications.<n>Retrieval-augmented generation (RAG) and conformal factuality have emerged as potential ways to address this limitation.<n>We analyze the reliability and usefulness of conformal factuality for RAG-based LLMs across generation, scoring, calibration, robustness, and efficiency.
  arXiv  Detail & Related papers  (2026-03-17T17:20:08Z)
• Calibratable Disambiguation Loss for Multi-Instance Partial-Label Learning [53.9713678229744]
  Multi-instance partial-label learning (MIPL) is a weakly supervised framework that addresses the challenges of inexact supervision in both instance and label spaces.<n>Existing MIPL approaches often suffer from poor calibration, undermining reliability.<n>We propose a plug-and-play calibratable disambiguation loss (CDL) that simultaneously improves classification accuracy and calibration performance.
  arXiv  Detail & Related papers  (2025-12-19T16:58:31Z)
• CoT-Saliency: Unified Chain-of-Thought Reasoning for Heterogeneous Saliency Tasks [96.64597365827046]
  We present the first unified framework that jointly handles three operationally heterogeneous saliency tasks.<n>We introduce a Chain-of-Thought (CoT) reasoning process in a Vision-Language Model (VLM) to bridge task heterogeneity.<n>We show our model matches or outperforms specialized SOTA methods and strong closed-source VLMs across all tasks.
  arXiv  Detail & Related papers  (2025-11-01T04:37:01Z)
• Certified Self-Consistency: Statistical Guarantees and Test-Time Training for Reliable Reasoning in LLMs [4.829906774017035]
  We present a unified framework for certifiable inference in large language models.<n>We show that majority voting provides a statistical certificate of self-consistency.<n>We further prove that label-free post-training methods such as TTRL implicitly sharpen the answer distribution.
  arXiv  Detail & Related papers  (2025-10-20T12:14:12Z)
• Judging with Confidence: Calibrating Autoraters to Preference Distributions [56.17041629492863]
  We argue that a reliable autorater must learn to model the full distribution of preferences defined by a target population.<n>We present two learning methods tailored to different data conditions.<n>Our results show that finetuning autoraters with a distribution-matching objective leads to verbalized probability predictions that are better aligned with the target preference distribution.
  arXiv  Detail & Related papers  (2025-09-30T20:36:41Z)
• A Comparative Benchmark of Large Language Models for Labelling Wind Turbine Maintenance Logs [0.0]
  This paper presents a framework for benchmarking Large Language Models (LLMs) on the task of classifying complex industrial records.<n>To promote transparency and encourage further research, this framework has been made publicly available as an open-source tool.<n>We quantify a clear performance hierarchy, identifying top models that exhibit high alignment with a benchmark standard and trustworthy, well-calibrated confidence scores.
  arXiv  Detail & Related papers  (2025-09-08T15:48:17Z)
• Semi-Supervised Federated Learning via Dual Contrastive Learning and Soft Labeling for Intelligent Fault Diagnosis [30.60728200709919]
  This paper proposes a semi-supervised federated learning framework, SSFL-DCSL.<n>It integrates dual contrastive loss and soft labeling to address data and label scarcity for distributed clients.<n>It can improve accuracy by 1.15% to 7.85% over state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-07-12T10:54:23Z)
• Binary Classification with Confidence Difference [100.08818204756093]
  This paper delves into a novel weakly supervised binary classification problem called confidence-difference (ConfDiff) classification. We propose a risk-consistent approach to tackle this problem and show that the estimation error bound the optimal convergence rate. We also introduce a risk correction approach to mitigate overfitting problems, whose consistency and convergence rate are also proven.
  arXiv  Detail & Related papers  (2023-10-09T11:44:50Z)
• From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning [52.257422715393574]
  We introduce a self-guided methodology for Large Language Models (LLMs) to autonomously discern and select cherry samples from open-source datasets. Our key innovation, the Instruction-Following Difficulty (IFD) metric, emerges as a pivotal metric to identify discrepancies between a model's expected responses and its intrinsic generation capability.
  arXiv  Detail & Related papers  (2023-08-23T09:45:29Z)
• InPL: Pseudo-labeling the Inliers First for Imbalanced Semi-supervised Learning [34.062061310242385]
  We present a new perspective of pseudo-labeling for imbalanced semi-supervised learning (SSL) We measure whether an unlabeled sample is likely to be in-distribution'' or out-of-distribution'' Experiments demonstrate that our energy-based pseudo-labeling method, textbfInPL, significantly outperforms confidence-based methods on imbalanced SSL benchmarks.
  arXiv  Detail & Related papers  (2023-03-13T16:45:41Z)
• SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learning [101.86916775218403]
  This paper revisits the popular pseudo-labeling methods via a unified sample weighting formulation. We propose SoftMatch to overcome the trade-off by maintaining both high quantity and high quality of pseudo-labels during training. In experiments, SoftMatch shows substantial improvements across a wide variety of benchmarks, including image, text, and imbalanced classification.
  arXiv  Detail & Related papers  (2023-01-26T03:53:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.