RoSearch: Search for Robust Student Architectures When Distilling Pre-trained Language Models

• URL: http://arxiv.org/abs/2106.03613v1
• Date: Mon, 7 Jun 2021 13:38:16 GMT
• Title: RoSearch: Search for Robust Student Architectures When Distilling Pre-trained Language Models
• Authors: Xin Guo, Jianlei Yang, Haoyi Zhou, Xucheng Ye, Jianxin Li
• Abstract summary: Student models acquired by knowledge distillation suffer from adversarial attacks, which limits their usage in security sensitive scenarios. A directed acyclic graph based search space is built and an evolutionary search strategy is utilized to guide the searching approach. Experimental results show that RoSearch can improve robustness of student models from 7%18% up to 45.8%47.8% on different datasets.
• Score: 11.906350752875342
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pre-trained language models achieve outstanding performance in NLP tasks. Various knowledge distillation methods have been proposed to reduce the heavy computation and storage requirements of pre-trained language models. However, from our observations, student models acquired by knowledge distillation suffer from adversarial attacks, which limits their usage in security sensitive scenarios. In order to overcome these security problems, RoSearch is proposed as a comprehensive framework to search the student models with better adversarial robustness when performing knowledge distillation. A directed acyclic graph based search space is built and an evolutionary search strategy is utilized to guide the searching approach. Each searched architecture is trained by knowledge distillation on pre-trained language model and then evaluated under a robustness-, accuracy- and efficiency-aware metric as environmental fitness. Experimental results show that RoSearch can improve robustness of student models from 7%~18% up to 45.8%~47.8% on different datasets with comparable weight compression ratio to existing distillation methods (4.6$\times$~6.5$\times$ improvement from teacher model BERT_BASE) and low accuracy drop. In addition, we summarize the relationship between student architecture and robustness through statistics of searched models.

Related papers

• AI Generated Text Detection [0.0]
  This paper presents an evaluation of AI text detection methods, including both traditional machine learning models and transformer-based architectures.<n>We utilize two datasets, HC3 and DAIGT v2, to build a unified benchmark and apply a topic-based data split to prevent information leakage.<n>Results indicate that contextual modeling is significantly superior to lexical features and highlight the importance of mitigating topic memorization.
  arXiv  Detail & Related papers  (2026-01-07T11:18:10Z)
• Think Before You Retrieve: Learning Test-Time Adaptive Search with Small Language Models [28.80331720382804]
  We introduce Orion, a training framework that enables compact models to perform iterative retrieval through learned search strategies.<n>Orion combines synthetic trajectory generation and supervised fine-tuning to encourage diverse exploration patterns in models.<n>Despite using only 3% of the training data available, our 1.2B model achieves 77.6% success on SciFact.
  arXiv  Detail & Related papers  (2025-11-10T19:49:55Z)
• Knowledge Distillation Detection for Open-weights Models [30.640429431816614]
  We introduce a model-agnostic framework that combines data-free input synthesis and statistical score for detecting distillation.<n>Our approach is applicable to both classification and generative models.
  arXiv  Detail & Related papers  (2025-10-02T17:59:14Z)
• Smooth-Distill: A Self-distillation Framework for Multitask Learning with Wearable Sensor Data [0.0]
  This paper introduces Smooth-Distill, a novel self-distillation framework designed to simultaneously perform human activity recognition (HAR) and sensor placement detection.<n>Unlike conventional distillation methods that require separate teacher and student models, the proposed framework utilizes a smoothed, historical version of the model itself as the teacher.<n> Experimental results show that Smooth-Distill consistently outperforms alternative approaches across different evaluation scenarios.
  arXiv  Detail & Related papers  (2025-06-27T06:51:51Z)
• Trace-of-Thought Prompting: Investigating Prompt-Based Knowledge Distillation Through Question Decomposition [6.066322919105025]
  We introduce Trace-of-Thought Prompting, a novel framework designed to distill critical reasoning capabilities from high-resource teacher models to low-resource student models.<n>Our results suggest a promising pathway for open-source, low-resource models to eventually serve both as both students and teachers.
  arXiv  Detail & Related papers  (2025-04-29T17:14:54Z)
• Honey, I Shrunk the Language Model: Impact of Knowledge Distillation Methods on Performance and Explainability [3.224880576815583]
  High computational and storage demands of Large Language Models limit their deployment in resource-constrained environments. Previous research has introduced several distillation methods for both generating training data and for training the student model. Despite their relevance, the effects of state-of-the-art distillation methods on model performance and explainability have not been thoroughly investigated.
  arXiv  Detail & Related papers  (2025-04-22T17:32:48Z)
• Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation [64.15918654558816]
  Self-distillation (SSD) training strategy is introduced for filtering and weighting teacher representation to distill from task-relevant representations only. Experimental results on real-world affective computing, wearable/biosignal datasets from the UCR Archive, the HAR dataset, and image classification datasets show that the proposed SSD method can outperform state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-04-19T14:08:56Z)
• The Surprising Effectiveness of Test-Time Training for Abstract Reasoning [64.36534512742736]
  We investigate the effectiveness of test-time training (TTT) as a mechanism for improving models' reasoning capabilities. TTT significantly improves performance on ARC tasks, achieving up to 6x improvement in accuracy compared to base fine-tuned models. Our findings suggest that explicit symbolic search is not the only path to improved abstract reasoning in neural language models.
  arXiv  Detail & Related papers  (2024-11-11T18:59:45Z)
• Zero-shot Retrieval: Augmenting Pre-trained Models with Search Engines [83.65380507372483]
  Large pre-trained models can dramatically reduce the amount of task-specific data required to solve a problem, but they often fail to capture domain-specific nuances out of the box. This paper shows how to leverage recent advances in NLP and multi-modal learning to augment a pre-trained model with search engine retrieval.
  arXiv  Detail & Related papers  (2023-11-29T05:33:28Z)
• One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation [69.65734716679925]
  Knowledge distillation has proven to be a highly effective approach for enhancing model performance through a teacher-student training scheme. Most existing distillation methods are designed under the assumption that the teacher and student models belong to the same model family. We propose a simple yet effective one-for-all KD framework called OFA-KD, which significantly improves the distillation performance between heterogeneous architectures.
  arXiv  Detail & Related papers  (2023-10-30T11:13:02Z)
• DisWOT: Student Architecture Search for Distillation WithOut Training [0.0]
  We explore a novel training-free framework to search for the best student architectures for a given teacher. Our work first empirically show that the optimal model under vanilla training cannot be the winner in distillation. Our experiments on CIFAR, ImageNet and NAS-Bench-201 demonstrate that our technique achieves state-of-the-art results on different search spaces.
  arXiv  Detail & Related papers  (2023-03-28T01:58:45Z)
• EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval [83.79667141681418]
  Large neural models (such as Transformers) achieve state-of-the-art performance for information retrieval (IR) We propose a novel distillation approach that leverages the relative geometry among queries and documents learned by the large teacher model. We show that our approach successfully distills from both dual-encoder (DE) and cross-encoder (CE) teacher models to 1/10th size asymmetric students that can retain 95-97% of the teacher performance.
  arXiv  Detail & Related papers  (2023-01-27T22:04:37Z)
• Context Unaware Knowledge Distillation for Image Retrieval [11.38957822323395]
  Existing knowledge distillation methods use logits and other features of the deep (teacher) model. We propose context unaware knowledge distillation that uses the knowledge of the teacher model without fine-tuning it on the target context.
  arXiv  Detail & Related papers  (2022-07-19T04:51:39Z)
• Robust Active Learning: Sample-Efficient Training of Robust Deep Learning Models [23.45276407731157]
  We propose emphrobust active learning, an active learning process that integrates adversarial training. We show that robust active learning can produce models with the robustness (accuracy on adversarial examples) ranging from 2.35% to 63.85%. Our acquisition function -- named density-based robust sampling with entropy (DRE) -- outperforms the other acquisition functions (including random) in terms of robustness.
  arXiv  Detail & Related papers  (2021-12-05T11:13:50Z)
• Voting based ensemble improves robustness of defensive models [82.70303474487105]
  We study whether it is possible to create an ensemble to further improve robustness. By ensembling several state-of-the-art pre-trained defense models, our method can achieve a 59.8% robust accuracy.
  arXiv  Detail & Related papers  (2020-11-28T00:08:45Z)
• Autoregressive Knowledge Distillation through Imitation Learning [70.12862707908769]
  We develop a compression technique for autoregressive models driven by an imitation learning perspective on knowledge distillation. Our method consistently outperforms other distillation algorithms, such as sequence-level knowledge distillation. Student models trained with our method attain 1.4 to 4.8 BLEU/ROUGE points higher than those trained from scratch, while increasing inference speed by up to 14 times in comparison to the teacher model.
  arXiv  Detail & Related papers  (2020-09-15T17:43:02Z)
• Adversarial Concurrent Training: Optimizing Robustness and Accuracy Trade-off of Deep Neural Networks [13.041607703862724]
  We propose Adversarial Concurrent Training (ACT) to train a robust model in conjunction with a natural model in a minimax game. ACT achieves 68.20% standard accuracy and 44.29% robustness accuracy under a 100-iteration untargeted attack.
  arXiv  Detail & Related papers  (2020-08-16T22:14:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.