Robust Training of Federated Models with Extremely Label Deficiency

• URL: http://arxiv.org/abs/2402.14430v1
• Date: Thu, 22 Feb 2024 10:19:34 GMT
• Title: Robust Training of Federated Models with Extremely Label Deficiency
• Authors: Yonggang Zhang, Zhiqin Yang, Xinmei Tian, Nannan Wang, Tongliang Liu, Bo Han
• Abstract summary: Federated semi-supervised learning (FSSL) has emerged as a powerful paradigm for collaboratively training machine learning models using distributed data with label deficiency. We propose a novel twin-model paradigm, called Twin-sight, designed to enhance mutual guidance by providing insights from different perspectives of labeled and unlabeled data. Our comprehensive experiments on four benchmark datasets provide substantial evidence that Twin-sight can significantly outperform state-of-the-art methods across various experimental settings.
• Score: 84.00832527512148
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated semi-supervised learning (FSSL) has emerged as a powerful paradigm for collaboratively training machine learning models using distributed data with label deficiency. Advanced FSSL methods predominantly focus on training a single model on each client. However, this approach could lead to a discrepancy between the objective functions of labeled and unlabeled data, resulting in gradient conflicts. To alleviate gradient conflict, we propose a novel twin-model paradigm, called Twin-sight, designed to enhance mutual guidance by providing insights from different perspectives of labeled and unlabeled data. In particular, Twin-sight concurrently trains a supervised model with a supervised objective function while training an unsupervised model using an unsupervised objective function. To enhance the synergy between these two models, Twin-sight introduces a neighbourhood-preserving constraint, which encourages the preservation of the neighbourhood relationship among data features extracted by both models. Our comprehensive experiments on four benchmark datasets provide substantial evidence that Twin-sight can significantly outperform state-of-the-art methods across various experimental settings, demonstrating the efficacy of the proposed Twin-sight.

Related papers

• Convergence Behavior of an Adversarial Weak Supervision Method [10.409652277630133]
  Weak Supervision is a paradigm subsuming subareas of machine learning. By using labeled data to train modern machine learning methods, the cost of acquiring large amounts of hand labeled data can be ameliorated. Two approaches to combining the rules-of-thumb falls into two camps, reflecting different ideologies of statistical estimation.
  arXiv  Detail & Related papers  (2024-05-25T02:33:17Z)
• Adversarial Augmentation Training Makes Action Recognition Models More Robust to Realistic Video Distribution Shifts [13.752169303624147]
  Action recognition models often lack robustness when faced with natural distribution shifts between training and test data. We propose two novel evaluation methods to assess model resilience to such distribution disparity. We experimentally demonstrate the superior performance of the proposed adversarial augmentation approach over baselines across three state-of-the-art action recognition models.
  arXiv  Detail & Related papers  (2024-01-21T05:50:39Z)
• Co-guiding for Multi-intent Spoken Language Understanding [53.30511968323911]
  We propose a novel model termed Co-guiding Net, which implements a two-stage framework achieving the mutual guidances between the two tasks. For the first stage, we propose single-task supervised contrastive learning, and for the second stage, we propose co-guiding supervised contrastive learning. Experiment results on multi-intent SLU show that our model outperforms existing models by a large margin.
  arXiv  Detail & Related papers  (2023-11-22T08:06:22Z)
• Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model [74.62272538148245]
  We show that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other. We investigate if it is possible to transfer such "complementary" knowledge from one model to another without performance degradation.
  arXiv  Detail & Related papers  (2023-10-26T17:59:46Z)
• Leveraging Diffusion Disentangled Representations to Mitigate Shortcuts in Underspecified Visual Tasks [92.32670915472099]
  We propose an ensemble diversification framework exploiting the generation of synthetic counterfactuals using Diffusion Probabilistic Models (DPMs) We show that diffusion-guided diversification can lead models to avert attention from shortcut cues, achieving ensemble diversity performance comparable to previous methods requiring additional data collection.
  arXiv  Detail & Related papers  (2023-10-03T17:37:52Z)
• Universal Semi-supervised Model Adaptation via Collaborative Consistency Training [92.52892510093037]
  We introduce a realistic and challenging domain adaptation problem called Universal Semi-supervised Model Adaptation (USMA) We propose a collaborative consistency training framework that regularizes the prediction consistency between two models. Experimental results demonstrate the effectiveness of our method on several benchmark datasets.
  arXiv  Detail & Related papers  (2023-07-07T08:19:40Z)
• Integrated Weak Learning [25.47289093245517]
  Integrated Weak Learning is a principled framework that integrates weak supervision into the training process of machine learning models. We show that our approach outperforms existing weak learning techniques across a set of 6 benchmark classification datasets.
  arXiv  Detail & Related papers  (2022-06-19T22:13:59Z)
• TOV: The Original Vision Model for Optical Remote Sensing Image Understanding via Self-supervised Learning [13.57667361338603]
  We propose textbfThe textbfOriginal textbfVision model (TOV) in remote sensing filed. Trained by massive unlabeled optical data along a human-like self-supervised learning path, TOV model can be easily adapted to various RSIU tasks. We analyze the influences of two key factors on the performance of building TOV model for RSIU, including the influence of using different data sampling methods.
  arXiv  Detail & Related papers  (2022-04-10T16:25:05Z)
• Adversarial Dual-Student with Differentiable Spatial Warping for Semi-Supervised Semantic Segmentation [70.2166826794421]
  We propose a differentiable geometric warping to conduct unsupervised data augmentation. We also propose a novel adversarial dual-student framework to improve the Mean-Teacher. Our solution significantly improves the performance and state-of-the-art results are achieved on both datasets.
  arXiv  Detail & Related papers  (2022-03-05T17:36:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.