Universal Metric Learning with Parameter-Efficient Transfer Learning
- URL: http://arxiv.org/abs/2309.08944v1
- Date: Sat, 16 Sep 2023 10:34:01 GMT
- Title: Universal Metric Learning with Parameter-Efficient Transfer Learning
- Authors: Sungyeon Kim, Donghyun Kim, Suha Kwak
- Abstract summary: A common practice in metric learning is to train and test an embedding model for each dataset.
This dataset-specific approach fails to simulate real-world scenarios that involve multiple heterogeneous distributions of data.
We introduce a novel metric learning paradigm, called Universal Metric Learning (UML), which learns a unified metric capable of capturing relations across multiple data distributions.
- Score: 40.85295050164728
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: A common practice in metric learning is to train and test an embedding model
for each dataset. This dataset-specific approach fails to simulate real-world
scenarios that involve multiple heterogeneous distributions of data. In this
regard, we introduce a novel metric learning paradigm, called Universal Metric
Learning (UML), which learns a unified distance metric capable of capturing
relations across multiple data distributions. UML presents new challenges, such
as imbalanced data distribution and bias towards dominant distributions. To
address these challenges, we propose Parameter-efficient Universal Metric
leArning (PUMA), which consists of a pre-trained frozen model and two
additional modules, stochastic adapter and prompt pool. These modules enable to
capture dataset-specific knowledge while avoiding bias towards dominant
distributions. Additionally, we compile a new universal metric learning
benchmark with a total of 8 different datasets. PUMA outperformed the
state-of-the-art dataset-specific models while using about 69 times fewer
trainable parameters.
Related papers
- Interpetable Target-Feature Aggregation for Multi-Task Learning based on Bias-Variance Analysis [53.38518232934096]
Multi-task learning (MTL) is a powerful machine learning paradigm designed to leverage shared knowledge across tasks to improve generalization and performance.
We propose an MTL approach at the intersection between task clustering and feature transformation based on a two-phase iterative aggregation of targets and features.
In both phases, a key aspect is to preserve the interpretability of the reduced targets and features through the aggregation with the mean, which is motivated by applications to Earth science.
arXiv Detail & Related papers (2024-06-12T08:30:16Z) - Decoupled Federated Learning on Long-Tailed and Non-IID data with
Feature Statistics [20.781607752797445]
We propose a two-stage Decoupled Federated learning framework using Feature Statistics (DFL-FS)
In the first stage, the server estimates the client's class coverage distributions through masked local feature statistics clustering.
In the second stage, DFL-FS employs federated feature regeneration based on global feature statistics to enhance the model's adaptability to long-tailed data distributions.
arXiv Detail & Related papers (2024-03-13T09:24:59Z) - Multiply Robust Estimation for Local Distribution Shifts with Multiple Domains [9.429772474335122]
We focus on scenarios where data distributions vary across multiple segments of the entire population.
We propose a two-stage multiply robust estimation method to improve model performance on each individual segment.
Our method is designed to be implemented with commonly used off-the-shelf machine learning models.
arXiv Detail & Related papers (2024-02-21T22:01:10Z) - Federated Learning with Projected Trajectory Regularization [65.6266768678291]
Federated learning enables joint training of machine learning models from distributed clients without sharing their local data.
One key challenge in federated learning is to handle non-identically distributed data across the clients.
We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
arXiv Detail & Related papers (2023-12-22T02:12:08Z) - FedSym: Unleashing the Power of Entropy for Benchmarking the Algorithms
for Federated Learning [1.4656078321003647]
Federated learning (FL) is a decentralized machine learning approach where independent learners process data privately.
We study the currently popular data partitioning techniques and visualize their main disadvantages.
We propose a method that leverages entropy and symmetry to construct 'the most challenging' and controllable data distributions.
arXiv Detail & Related papers (2023-10-11T18:39:08Z) - Adaptive Parameterization of Deep Learning Models for Federated Learning [85.82002651944254]
Federated Learning offers a way to train deep neural networks in a distributed fashion.
It incurs a communication overhead as the model parameters or gradients need to be exchanged regularly during training.
In this paper, we propose to utilise parallel Adapters for Federated Learning.
arXiv Detail & Related papers (2023-02-06T17:30:33Z) - Learning from aggregated data with a maximum entropy model [73.63512438583375]
We show how a new model, similar to a logistic regression, may be learned from aggregated data only by approximating the unobserved feature distribution with a maximum entropy hypothesis.
We present empirical evidence on several public datasets that the model learned this way can achieve performances comparable to those of a logistic model trained with the full unaggregated data.
arXiv Detail & Related papers (2022-10-05T09:17:27Z) - Meta-Causal Feature Learning for Out-of-Distribution Generalization [71.38239243414091]
This paper presents a balanced meta-causal learner (BMCL), which includes a balanced task generation module (BTG) and a meta-causal feature learning module (MCFL)
BMCL effectively identifies the class-invariant visual regions for classification and may serve as a general framework to improve the performance of the state-of-the-art methods.
arXiv Detail & Related papers (2022-08-22T09:07:02Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.