Diversified Mutual Learning for Deep Metric Learning
- URL: http://arxiv.org/abs/2009.04170v1
- Date: Wed, 9 Sep 2020 09:00:16 GMT
- Title: Diversified Mutual Learning for Deep Metric Learning
- Authors: Wonpyo Park, Wonjae Kim, Kihyun You, Minsu Cho
- Abstract summary: Mutual learning is an ensemble training strategy to improve generalization.
We propose an effective mutual learning method for deep metric learning, called Diversified Mutual Metric Learning.
Our method significantly improves individual models as well as their ensemble.
- Score: 42.42997713655545
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Mutual learning is an ensemble training strategy to improve generalization by
transferring individual knowledge to each other while simultaneously training
multiple models. In this work, we propose an effective mutual learning method
for deep metric learning, called Diversified Mutual Metric Learning, which
enhances embedding models with diversified mutual learning. We transfer
relational knowledge for deep metric learning by leveraging three kinds of
diversities in mutual learning: (1) model diversity from different
initializations of models, (2) temporal diversity from different frequencies of
parameter update, and (3) view diversity from different augmentations of
inputs. Our method is particularly adequate for inductive transfer learning at
the lack of large-scale data, where the embedding model is initialized with a
pretrained model and then fine-tuned on a target dataset. Extensive experiments
show that our method significantly improves individual models as well as their
ensemble. Finally, the proposed method with a conventional triplet loss
achieves the state-of-the-art performance of Recall@1 on standard datasets:
69.9 on CUB-200-2011 and 89.1 on CARS-196.
Related papers
- Cross-Training with Multi-View Knowledge Fusion for Heterogenous Federated Learning [13.796783869133531]
This paper presents a novel approach that enhances federated learning through a cross-training scheme incorporating multi-view information.
Specifically, the proposed method, termed FedCT, includes three main modules, where the consistency-aware knowledge broadcasting module aims to optimize model assignment strategies.
The multi-view knowledge-guided representation learning module leverages fused knowledge from both global and local views to enhance the preservation of local knowledge before and after model exchange.
The mixup-based feature augmentation module aggregates rich information to further increase the diversity of feature spaces, which enables the model to better discriminate complex samples.
arXiv Detail & Related papers (2024-05-30T13:27:30Z) - Personalized Federated Learning with Contextual Modulation and
Meta-Learning [2.7716102039510564]
Federated learning has emerged as a promising approach for training machine learning models on decentralized data sources.
We propose a novel framework that combines federated learning with meta-learning techniques to enhance both efficiency and generalization capabilities.
arXiv Detail & Related papers (2023-12-23T08:18:22Z) - Fantastic Gains and Where to Find Them: On the Existence and Prospect of
General Knowledge Transfer between Any Pretrained Model [74.62272538148245]
We show that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other.
We investigate if it is possible to transfer such "complementary" knowledge from one model to another without performance degradation.
arXiv Detail & Related papers (2023-10-26T17:59:46Z) - Learning Unseen Modality Interaction [54.23533023883659]
Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences.
We pose the problem of unseen modality interaction and introduce a first solution.
It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved.
arXiv Detail & Related papers (2023-06-22T10:53:10Z) - An Empirical Study of Multimodal Model Merging [148.48412442848795]
Model merging is a technique that fuses multiple models trained on different tasks to generate a multi-task solution.
We conduct our study for a novel goal where we can merge vision, language, and cross-modal transformers of a modality-specific architecture.
We propose two metrics that assess the distance between weights to be merged and can serve as an indicator of the merging outcomes.
arXiv Detail & Related papers (2023-04-28T15:43:21Z) - Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models.
This creates a barrier to fusing knowledge across individual models to yield a better single model.
We propose a dataless knowledge fusion method that merges models in their parameter space.
arXiv Detail & Related papers (2022-12-19T20:46:43Z) - Leveraging Different Learning Styles for Improved Knowledge Distillation
in Biomedical Imaging [0.9208007322096533]
Our work endeavors to leverage the concept of knowledge diversification to improve the performance of model compression techniques like Knowledge Distillation (KD) and Mutual Learning (ML)
We use a single-teacher and two-student network in a unified framework that not only allows for the transfer of knowledge from teacher to students (KD) but also encourages collaborative learning between students (ML)
Unlike the conventional approach, where the teacher shares the same knowledge in the form of predictions or feature representations with the student network, our proposed approach employs a more diversified strategy by training one student with predictions and the other with feature maps from the teacher.
arXiv Detail & Related papers (2022-12-06T12:40:45Z) - Integrating Contrastive Learning with Dynamic Models for Reinforcement
Learning from Images [31.413588478694496]
We argue that explicitly improving Markovianity of the learned embedding is desirable.
We propose a self-supervised representation learning method which integrates contrastive learning with dynamic models.
arXiv Detail & Related papers (2022-03-02T14:39:17Z) - Merging Models with Fisher-Weighted Averaging [24.698591753644077]
We introduce a fundamentally different method for transferring knowledge across models that amounts to "merging" multiple models into one.
Our approach effectively involves computing a weighted average of the models' parameters.
We show that our merging procedure makes it possible to combine models in previously unexplored ways.
arXiv Detail & Related papers (2021-11-18T17:59:35Z) - Emerging Trends in Federated Learning: From Model Fusion to Federated X Learning [65.06445195580622]
Federated learning is a new paradigm that decouples data collection and model training via multi-party computation and model aggregation.
We conduct a focused survey of federated learning in conjunction with other learning algorithms.
arXiv Detail & Related papers (2021-02-25T15:18:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.