Related papers: On Good Practices for Task-Specific Distillation of Large Pretrained Visual Models

On Good Practices for Task-Specific Distillation of Large Pretrained Visual Models

URL: http://arxiv.org/abs/2402.11305v2
Date: Tue, 7 May 2024 15:30:45 GMT
Title: On Good Practices for Task-Specific Distillation of Large Pretrained Visual Models
Authors: Juliette Marrie, Michael Arbel, Julien Mairal, Diane Larlus,
Abstract summary: We show that the excellent robustness and versatility of recent pretrained models challenge common practices established in the literature. We also show that a variant of Mixup based on stable diffusion complements standard data augmentation.
Score: 42.57860180847724
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Large pretrained visual models exhibit remarkable generalization across diverse recognition tasks. Yet, real-world applications often demand compact models tailored to specific problems. Variants of knowledge distillation have been devised for such a purpose, enabling task-specific compact models (the students) to learn from a generic large pretrained one (the teacher). In this paper, we show that the excellent robustness and versatility of recent pretrained models challenge common practices established in the literature, calling for a new set of optimal guidelines for task-specific distillation. To address the lack of samples in downstream tasks, we also show that a variant of Mixup based on stable diffusion complements standard data augmentation. This strategy eliminates the need for engineered text prompts and improves distillation of generic models into streamlined specialized networks.

Related papers

Faithful Label-free Knowledge Distillation [8.572967695281054]
This paper presents a label-free knowledge distillation approach called Teacher in the Middle (TinTeM) It produces a more faithful student, which better replicates the behavior of the teacher network across a range of benchmarks testing model robustness, generalisability and out-of-distribution detection.
arXiv Detail & Related papers (2024-11-22T01:48:44Z)
Diversified Batch Selection for Training Acceleration [68.67164304377732]
A prevalent research line, known as online batch selection, explores selecting informative subsets during the training process. vanilla reference-model-free methods involve independently scoring and selecting data in a sample-wise manner. We propose Diversified Batch Selection (DivBS), which is reference-model-free and can efficiently select diverse and representative samples.
arXiv Detail & Related papers (2024-06-07T12:12:20Z)
Data-efficient Large Vision Models through Sequential Autoregression [58.26179273091461]
We develop an efficient, autoregression-based vision model on a limited dataset. We demonstrate how this model achieves proficiency in a spectrum of visual tasks spanning both high-level and low-level semantic understanding. Our empirical evaluations underscore the model's agility in adapting to various tasks, heralding a significant reduction in the parameter footprint.
arXiv Detail & Related papers (2024-02-07T13:41:53Z)
Bridging Generative and Discriminative Models for Unified Visual Perception with Diffusion Priors [56.82596340418697]
We propose a simple yet effective framework comprising a pre-trained Stable Diffusion (SD) model containing rich generative priors, a unified head (U-head) capable of integrating hierarchical representations, and an adapted expert providing discriminative priors. Comprehensive investigations unveil potential characteristics of Vermouth, such as varying granularity of perception concealed in latent variables at distinct time steps and various U-net stages. The promising results demonstrate the potential of diffusion models as formidable learners, establishing their significance in furnishing informative and robust visual representations.
arXiv Detail & Related papers (2024-01-29T10:36:57Z)
LatentDR: Improving Model Generalization Through Sample-Aware Latent Degradation and Restoration [22.871920291497094]
We propose a novel approach for distribution-aware latent augmentation. Our approach first degrades the samples in the latent space, mapping them to augmented labels, and then restores the samples during training. We show that our method can be flexibly adapted to long-tail recognition tasks, demonstrating its versatility in building more generalizable models.
arXiv Detail & Related papers (2023-08-28T14:08:42Z)
Prototype-guided Cross-task Knowledge Distillation for Large-scale Models [103.04711721343278]
Cross-task knowledge distillation helps to train a small student model to obtain a competitive performance. We propose a Prototype-guided Cross-task Knowledge Distillation (ProC-KD) approach to transfer the intrinsic local-level object knowledge of a large-scale teacher network to various task scenarios.
arXiv Detail & Related papers (2022-12-26T15:00:42Z)
Squared $\ell_2$ Norm as Consistency Loss for Leveraging Augmented Data to Learn Robust and Invariant Representations [76.85274970052762]
Regularizing distance between embeddings/representations of original samples and augmented counterparts is a popular technique for improving robustness of neural networks. In this paper, we explore these various regularization choices, seeking to provide a general understanding of how we should regularize the embeddings. We show that the generic approach we identified (squared $ell$ regularized augmentation) outperforms several recent methods, which are each specially designed for one task.
arXiv Detail & Related papers (2020-11-25T22:40:09Z)
Manifold attack [0.22419496088582863]
In this paper, we enforce the manifold preservation (manifold learning) from the original data into latent presentation. We show that our approach of regularization provides improvements for the accuracy rate and for the robustness to adversarial examples.
arXiv Detail & Related papers (2020-09-13T09:39:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.