Sentinel-Guided Zero-Shot Learning: A Collaborative Paradigm without Real Data Exposure

• URL: http://arxiv.org/abs/2403.09363v1
• Date: Thu, 14 Mar 2024 13:12:49 GMT
• Title: Sentinel-Guided Zero-Shot Learning: A Collaborative Paradigm without Real Data Exposure
• Abstract summary: SG-ZSL is designed to foster efficient collaboration without the need to exchange models or sensitive data. It consists of a teacher model, a student model and a generator that links both model entities. It consistently outperforms in ZSL and GZSL tasks, notably in the white-box protocol.
• Score: 13.971962047728237
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With increasing concerns over data privacy and model copyrights, especially in the context of collaborations between AI service providers and data owners, an innovative SG-ZSL paradigm is proposed in this work. SG-ZSL is designed to foster efficient collaboration without the need to exchange models or sensitive data. It consists of a teacher model, a student model and a generator that links both model entities. The teacher model serves as a sentinel on behalf of the data owner, replacing real data, to guide the student model at the AI service provider's end during training. Considering the disparity of knowledge space between the teacher and student, we introduce two variants of the teacher model: the omniscient and the quasi-omniscient teachers. Under these teachers' guidance, the student model seeks to match the teacher model's performance and explores domains that the teacher has not covered. To trade off between privacy and performance, we further introduce two distinct security-level training protocols: white-box and black-box, enhancing the paradigm's adaptability. Despite the inherent challenges of real data absence in the SG-ZSL paradigm, it consistently outperforms in ZSL and GZSL tasks, notably in the white-box protocol. Our comprehensive evaluation further attests to its robustness and efficiency across various setups, including stringent black-box training protocol.

Related papers

• UCO: A Multi-Turn Interactive Reinforcement Learning Method for Adaptive Teaching with Large Language Models [59.693733170193944]
  Large language models (LLMs) are shifting from answer providers to intelligent tutors in educational settings.<n>Recent reinforcement learning approaches address this limitation but face two critical challenges.<n>We propose the Unidirectional Cognitive Optimization (UCO) method to address these challenges.
  arXiv  Detail & Related papers  (2025-11-12T01:27:02Z)
• Rethinking the Mean Teacher Strategy from the Perspective of Self-paced Learning [5.6818939992896365]
  Semi-supervised medical image segmentation has attracted significant attention due to its potential to reduce manual annotation costs.<n>In this work, we reinterpret the MT strategy on supervised data as a form of self-paced learning, regulated by the output agreement between the temporally lagged teacher model and the ground truth labels.
  arXiv  Detail & Related papers  (2025-05-16T09:14:06Z)
• Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling [81.00825302340984]
  We introduce Speculative Knowledge Distillation (SKD) to generate high-quality training data on-the-fly. In SKD, the student proposes tokens, and the teacher replaces poorly ranked ones based on its own distribution. We evaluate SKD on various text generation tasks, including translation, summarization, math, and instruction following.
  arXiv  Detail & Related papers  (2024-10-15T06:51:25Z)
• Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models [62.5501109475725]
  Knowledge distillation (KD) is a technique that compresses large teacher models by training smaller student models to mimic them. This paper introduces Online Knowledge Distillation (OKD), where the teacher network integrates small online modules to concurrently train with the student model. OKD achieves or exceeds the performance of leading methods in various model architectures and sizes, reducing training time by up to fourfold.
  arXiv  Detail & Related papers  (2024-09-19T07:05:26Z)
• CLDA: Collaborative Learning for Enhanced Unsupervised Domain Adaptation [15.97351561456467]
  Collaborative Learning is a method that updates the teacher's non-salient parameters using the student model and at the same time enhance the student's performance. CLDA achieves an improvement of +0.7% mIoU for teacher and +1.4% mIoU for student compared to the baseline model in the GTA to Cityscapes.
  arXiv  Detail & Related papers  (2024-09-04T13:35:15Z)
• Interactive DualChecker for Mitigating Hallucinations in Distilling Large Language Models [7.632217365130212]
  Large Language Models (LLMs) have demonstrated exceptional capabilities across various machine learning (ML) tasks. These models can produce hallucinations, particularly in domains with incomplete knowledge. We introduce DualChecker, an innovative framework designed to mitigate hallucinations and improve the performance of both teacher and student models.
  arXiv  Detail & Related papers  (2024-08-22T12:04:04Z)
• DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Driving [64.57963116462757]
  State-of-the-art methods usually follow the Teacher-Student' paradigm. Student model only has access to raw sensor data and conducts behavior cloning on the data collected by the teacher model. We propose DriveAdapter, which employs adapters with the feature alignment objective function between the student (perception) and teacher (planning) modules.
  arXiv  Detail & Related papers  (2023-08-01T09:21:53Z)
• Distantly-Supervised Named Entity Recognition with Adaptive Teacher Learning and Fine-grained Student Ensemble [56.705249154629264]
  Self-training teacher-student frameworks are proposed to improve the robustness of NER models. In this paper, we propose an adaptive teacher learning comprised of two teacher-student networks. Fine-grained student ensemble updates each fragment of the teacher model with a temporal moving average of the corresponding fragment of the student, which enhances consistent predictions on each model fragment against noise.
  arXiv  Detail & Related papers  (2022-12-13T12:14:09Z)
• Private Semi-supervised Knowledge Transfer for Deep Learning from Noisy Labels [21.374707481630697]
  We propose PATE++, which combines the current advanced noisy label training mechanisms with the original PATE framework to enhance its accuracy. A novel structure of Generative Adversarial Nets (GANs) is developed in order to integrate them effectively. In addition, we develop a novel noisy label detection mechanism for semi-supervised model training to further improve student model performance when training with noisy labels.
  arXiv  Detail & Related papers  (2022-11-03T07:33:49Z)
• Absolute Zero-Shot Learning [14.187187391391882]
  The key innovation is to involve a teacher model as the data safeguard to guide the AZSL model training without data leaking. We investigate black-box' and white-box' scenarios in AZSL task as different levels of model security. Our framework can retain state-of-the-art ZSL and GZSL performance under the white-box' scenario.
  arXiv  Detail & Related papers  (2022-02-23T05:33:21Z)
• Decentralized Federated Learning Preserves Model and Data Privacy [77.454688257702]
  We propose a fully decentralized approach, which allows to share knowledge between trained models. Students are trained on the output of their teachers via synthetically generated input data. The results show that an untrained student model, trained on the teachers output reaches comparable F1-scores as the teacher.
  arXiv  Detail & Related papers  (2021-02-01T14:38:54Z)
• FaceLeaks: Inference Attacks against Transfer Learning Models via Black-box Queries [2.7564955518050693]
  We investigate if one can leak or infer private information without interacting with the teacher model directly. We propose novel strategies to infer from aggregate-level information. Our study indicates that information leakage is a real privacy threat to the transfer learning framework widely used in real-life situations.
  arXiv  Detail & Related papers  (2020-10-27T03:02:40Z)
• Differentially Private Deep Learning with Smooth Sensitivity [144.31324628007403]
  We study privacy concerns through the lens of differential privacy. In this framework, privacy guarantees are generally obtained by perturbing models in such a way that specifics of data used to train the model are made ambiguous. One of the most important techniques used in previous works involves an ensemble of teacher models, which return information to a student based on a noisy voting procedure. In this work, we propose a novel voting mechanism with smooth sensitivity, which we call Immutable Noisy ArgMax, that, under certain conditions, can bear very large random noising from the teacher without affecting the useful information transferred to the student
  arXiv  Detail & Related papers  (2020-03-01T15:38:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.