Knowledge Distillation Methods for Efficient Unsupervised Adaptation Across Multiple Domains

• URL: http://arxiv.org/abs/2101.07308v1
• Date: Mon, 18 Jan 2021 19:53:16 GMT
• Title: Knowledge Distillation Methods for Efficient Unsupervised Adaptation Across Multiple Domains
• Authors: Le Thanh Nguyen-Meidine, Atif Belal, Madhu Kiran, Jose Dolz, Louis-Antoine Blais-Morin, Eric Granger
• Abstract summary: We propose a progressive KD approach for unsupervised single-target DA (STDA) and multi-target DA (MTDA) of CNNs. Our proposed approach is compared against state-of-the-art methods for compression and STDA of CNNs on the Office31 and ImageClef-DA image classification datasets.
• Score: 13.464493273131591
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Beyond the complexity of CNNs that require training on large annotated datasets, the domain shift between design and operational data has limited the adoption of CNNs in many real-world applications. For instance, in person re-identification, videos are captured over a distributed set of cameras with non-overlapping viewpoints. The shift between the source (e.g. lab setting) and target (e.g. cameras) domains may lead to a significant decline in recognition accuracy. Additionally, state-of-the-art CNNs may not be suitable for such real-time applications given their computational requirements. Although several techniques have recently been proposed to address domain shift problems through unsupervised domain adaptation (UDA), or to accelerate/compress CNNs through knowledge distillation (KD), we seek to simultaneously adapt and compress CNNs to generalize well across multiple target domains. In this paper, we propose a progressive KD approach for unsupervised single-target DA (STDA) and multi-target DA (MTDA) of CNNs. Our method for KD-STDA adapts a CNN to a single target domain by distilling from a larger teacher CNN, trained on both target and source domain data in order to maintain its consistency with a common representation. Our proposed approach is compared against state-of-the-art methods for compression and STDA of CNNs on the Office31 and ImageClef-DA image classification datasets. It is also compared against state-of-the-art methods for MTDA on Digits, Office31, and OfficeHome. In both settings -- KD-STDA and KD-MTDA -- results indicate that our approach can achieve the highest level of accuracy across target domains, while requiring a comparable or lower CNN complexity.

Related papers

• Open-Set Domain Adaptation with Visual-Language Foundation Models [51.49854335102149]
  Unsupervised domain adaptation (UDA) has proven to be very effective in transferring knowledge from a source domain to a target domain with unlabeled data. Open-set domain adaptation (ODA) has emerged as a potential solution to identify these classes during the training phase.
  arXiv  Detail & Related papers  (2023-07-30T11:38:46Z)
• Knowledge Distillation for Multi-Target Domain Adaptation in Real-Time Person Re-Identification [10.672152844970151]
  Multi-target domain adaptation (MTDA) has not been widely addressed in the ReID literature. We introduce a new MTDA method based on knowledge distillation (KD-ReID) that is suitable for real-time person ReID applications. Our method adapts a common lightweight student backbone CNN over the target domains by alternatively distilling from multiple specialized teacher CNNs, each one adapted on data from a specific target domain.
  arXiv  Detail & Related papers  (2022-05-12T17:28:02Z)
• Continual BatchNorm Adaptation (CBNA) for Semantic Segmentation [39.99513327031499]
  Environment perception in autonomous driving vehicles often heavily relies on deep neural networks (DNNs) In this work, we expand a source-free UDA approach to a continual and therefore online-capable UDA on a single-image basis for semantic segmentation. Our method Continual BatchNorm Adaptation (CBNA) modifies the source domain statistics in the batch normalization layers, using target domain images in an unsupervised fashion, which yields consistent performance improvements during inference.
  arXiv  Detail & Related papers  (2022-03-02T12:55:10Z)
• Stagewise Unsupervised Domain Adaptation with Adversarial Self-Training for Road Segmentation of Remote Sensing Images [93.50240389540252]
  Road segmentation from remote sensing images is a challenging task with wide ranges of application potentials. We propose a novel stagewise domain adaptation model called RoadDA to address the domain shift (DS) issue in this field. Experiment results on two benchmarks demonstrate that RoadDA can efficiently reduce the domain gap and outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-08-28T09:29:14Z)
• Incremental Multi-Target Domain Adaptation for Object Detection with Efficient Domain Transfer [15.000304613769108]
  Techniques for multi-target domain adaptation (MTDA) seek to adapt a recognition model such that it can generalize well across multiple target domains. Key challenges include the lack of bounding box annotations for target data, knowledge corruption, and the growing resource requirements needed to train accurate deep detection models. We propose a new Incremental MTDA technique for object detection that can adapt a detector to multiple target domains, one at a time, without having to retain data of previously-learned target domains.
  arXiv  Detail & Related papers  (2021-04-13T19:35:54Z)
• Instance Level Affinity-Based Transfer for Unsupervised Domain Adaptation [74.71931918541748]
  We propose an instance affinity based criterion for source to target transfer during adaptation, called ILA-DA. We first propose a reliable and efficient method to extract similar and dissimilar samples across source and target, and utilize a multi-sample contrastive loss to drive the domain alignment process. We verify the effectiveness of ILA-DA by observing consistent improvements in accuracy over popular domain adaptation approaches on a variety of benchmark datasets.
  arXiv  Detail & Related papers  (2021-04-03T01:33:14Z)
• BreakingBED -- Breaking Binary and Efficient Deep Neural Networks by Adversarial Attacks [65.2021953284622]
  We study robustness of CNNs against white-box and black-box adversarial attacks. Results are shown for distilled CNNs, agent-based state-of-the-art pruned models, and binarized neural networks.
  arXiv  Detail & Related papers  (2021-03-14T20:43:19Z)
• Background Adaptive Faster R-CNN for Semi-Supervised Convolutional Object Detection of Threats in X-Ray Images [64.39996451133268]
  We present a semi-supervised approach for threat recognition which we call Background Adaptive Faster R-CNN. This approach is a training method for two-stage object detectors which uses Domain Adaptation methods from the field of deep learning. Two domain discriminators, one for discriminating object proposals and one for image features, are adversarially trained to prevent encoding domain-specific information. This can reduce threat detection false alarm rates by matching the statistics of extracted features from hand-collected backgrounds to real world data.
  arXiv  Detail & Related papers  (2020-10-02T21:05:13Z)
• Deep Co-Training with Task Decomposition for Semi-Supervised Domain Adaptation [80.55236691733506]
  Semi-supervised domain adaptation (SSDA) aims to adapt models trained from a labeled source domain to a different but related target domain. We propose to explicitly decompose the SSDA task into two sub-tasks: a semi-supervised learning (SSL) task in the target domain and an unsupervised domain adaptation (UDA) task across domains.
  arXiv  Detail & Related papers  (2020-07-24T17:57:54Z)
• Keep it Simple: Image Statistics Matching for Domain Adaptation [0.0]
  Domain Adaptation (DA) is a technique to maintain detection accuracy when only unlabeled images are available of the target domain. Recent state-of-the-art methods try to reduce the domain gap using an adversarial training strategy. We propose to align either color histograms or mean and covariance of the source images towards the target domain. In comparison to recent methods, we achieve state-of-the-art performance using a much simpler procedure for the training.
  arXiv  Detail & Related papers  (2020-05-26T07:32:09Z)
• Joint Progressive Knowledge Distillation and Unsupervised Domain Adaptation [15.115086812609182]
  We propose an unexplored direction -- the joint optimization of CNNs to provide a compressed model that is adapted to perform well for a given target domain. Our method is compared against state-of-the-art compression and UDA techniques, using two popular classification datasets for UDA -- Office31 and ImageClef-DA.
  arXiv  Detail & Related papers  (2020-05-16T01:07:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.