Bridging Domains with Approximately Shared Features
- URL: http://arxiv.org/abs/2403.06424v1
- Date: Mon, 11 Mar 2024 04:25:41 GMT
- Title: Bridging Domains with Approximately Shared Features
- Authors: Ziliang Samuel Zhong, Xiang Pan, Qi Lei
- Abstract summary: Multi-source domain adaptation aims to reduce performance degradation when applying machine learning models to unseen domains.
Some advocate for learning invariant features from source domains, while others favor more diverse features.
We propose a statistical framework that distinguishes the utilities of features based on the variance of their correlation to label $y$ across domains.
- Score: 26.096779584142986
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Multi-source domain adaptation aims to reduce performance degradation when
applying machine learning models to unseen domains. A fundamental challenge is
devising the optimal strategy for feature selection. Existing literature is
somewhat paradoxical: some advocate for learning invariant features from source
domains, while others favor more diverse features. To address the challenge, we
propose a statistical framework that distinguishes the utilities of features
based on the variance of their correlation to label $y$ across domains. Under
our framework, we design and analyze a learning procedure consisting of
learning approximately shared feature representation from source tasks and
fine-tuning it on the target task. Our theoretical analysis necessitates the
importance of learning approximately shared features instead of only the
strictly invariant features and yields an improved population risk compared to
previous results on both source and target tasks, thus partly resolving the
paradox mentioned above. Inspired by our theory, we proposed a more practical
way to isolate the content (invariant+approximately shared) from environmental
features and further consolidate our theoretical findings.
Related papers
- Causality-inspired Latent Feature Augmentation for Single Domain Generalization [13.735443005394773]
Single domain generalization (Single-DG) intends to develop a generalizable model with only one single training domain to perform well on other unknown target domains.
Under the domain-hungry configuration, how to expand the coverage of source domain and find intrinsic causal features across different distributions is the key to enhancing the models' generalization ability.
We propose a novel causality-inspired latent feature augmentation method for Single-DG by learning the meta-knowledge of feature-level transformation based on causal learning and interventions.
arXiv Detail & Related papers (2024-06-10T02:42:25Z) - Generalization Performance of Transfer Learning: Overparameterized and
Underparameterized Regimes [61.22448274621503]
In real-world applications, tasks often exhibit partial similarity, where certain aspects are similar while others are different or irrelevant.
Our study explores various types of transfer learning, encompassing two options for parameter transfer.
We provide practical guidelines for determining the number of features in the common and task-specific parts for improved generalization performance.
arXiv Detail & Related papers (2023-06-08T03:08:40Z) - Multi-task Bias-Variance Trade-off Through Functional Constraints [102.64082402388192]
Multi-task learning aims to acquire a set of functions that perform well for diverse tasks.
In this paper we draw intuition from the two extreme learning scenarios -- a single function for all tasks, and a task-specific function that ignores the other tasks.
We introduce a constrained learning formulation that enforces domain specific solutions to a central function.
arXiv Detail & Related papers (2022-10-27T16:06:47Z) - Overcoming Shortcut Learning in a Target Domain by Generalizing Basic
Visual Factors from a Source Domain [7.012240324005977]
Shortcut learning occurs when a deep neural network overly relies on spurious correlations in the training dataset to solve downstream tasks.
We propose a novel approach to mitigate shortcut learning in uncontrolled target domains.
arXiv Detail & Related papers (2022-07-20T16:05:32Z) - Variational Disentanglement for Domain Generalization [68.85458536180437]
We propose to tackle the problem of domain generalization by delivering an effective framework named Variational Disentanglement Network (VDN)
VDN is capable of disentangling the domain-specific features and task-specific features, where the task-specific features are expected to be better generalized to unseen but related test data.
arXiv Detail & Related papers (2021-09-13T09:55:32Z) - On the Fundamental Trade-offs in Learning Invariant Representations [7.868449549351487]
We identify and determine two fundamental trade-offs between utility and semantic dependence induced by the statistical dependencies between the data and its corresponding target and semantic attributes.
We numerically quantify the trade-offs on representative problems and compare to the solutions achieved by baseline representation learning algorithms.
arXiv Detail & Related papers (2021-09-08T01:26:46Z) - Ensemble Making Few-Shot Learning Stronger [4.17701749612924]
This paper explores an ensemble approach to reduce the variance and introduces fine-tuning and feature attention strategies to calibrate relation-level features.
Results on several few-shot relation learning tasks show that our model significantly outperforms the previous state-of-the-art models.
arXiv Detail & Related papers (2021-05-12T17:11:10Z) - Fundamental Limits and Tradeoffs in Invariant Representation Learning [99.2368462915979]
Many machine learning applications involve learning representations that achieve two competing goals.
Minimax game-theoretic formulation represents a fundamental tradeoff between accuracy and invariance.
We provide an information-theoretic analysis of this general and important problem under both classification and regression settings.
arXiv Detail & Related papers (2020-12-19T15:24:04Z) - Learning to Combine: Knowledge Aggregation for Multi-Source Domain
Adaptation [56.694330303488435]
We propose a Learning to Combine for Multi-Source Domain Adaptation (LtC-MSDA) framework.
In the nutshell, a knowledge graph is constructed on the prototypes of various domains to realize the information propagation among semantically adjacent representations.
Our approach outperforms existing methods with a remarkable margin.
arXiv Detail & Related papers (2020-07-17T07:52:44Z) - Task-Feature Collaborative Learning with Application to Personalized
Attribute Prediction [166.87111665908333]
We propose a novel multi-task learning method called Task-Feature Collaborative Learning (TFCL)
Specifically, we first propose a base model with a heterogeneous block-diagonal structure regularizer to leverage the collaborative grouping of features and tasks.
As a practical extension, we extend the base model by allowing overlapping features and differentiating the hard tasks.
arXiv Detail & Related papers (2020-04-29T02:32:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.