Source data selection for out-of-domain generalization
- URL: http://arxiv.org/abs/2202.02155v1
- Date: Fri, 4 Feb 2022 14:37:31 GMT
- Title: Source data selection for out-of-domain generalization
- Authors: Xinran Miao and Kris Sankaran
- Abstract summary: Poor selection of a source dataset can lead to poor performance on the target.
We propose two source selection methods that are based on the multi-bandit theory and random search.
Our proposals can be viewed as diagnostics for the existence of a reweighted source subsamples that perform better than the random selection of available samples.
- Score: 0.76146285961466
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Models that perform out-of-domain generalization borrow knowledge from
heterogeneous source data and apply it to a related but distinct target task.
Transfer learning has proven effective for accomplishing this generalization in
many applications. However, poor selection of a source dataset can lead to poor
performance on the target, a phenomenon called negative transfer. In order to
take full advantage of available source data, this work studies source data
selection with respect to a target task. We propose two source selection
methods that are based on the multi-bandit theory and random search,
respectively. We conduct a thorough empirical evaluation on both simulated and
real data. Our proposals can be also viewed as diagnostics for the existence of
a reweighted source subsamples that perform better than the random selection of
available samples.
Related papers
- Diversified Batch Selection for Training Acceleration [68.67164304377732]
A prevalent research line, known as online batch selection, explores selecting informative subsets during the training process.
vanilla reference-model-free methods involve independently scoring and selecting data in a sample-wise manner.
We propose Diversified Batch Selection (DivBS), which is reference-model-free and can efficiently select diverse and representative samples.
arXiv Detail & Related papers (2024-06-07T12:12:20Z) - DsDm: Model-Aware Dataset Selection with Datamodels [81.01744199870043]
Standard practice is to filter for examples that match human notions of data quality.
We find that selecting according to similarity with "high quality" data sources may not increase (and can even hurt) performance compared to randomly selecting data.
Our framework avoids handpicked notions of data quality, and instead models explicitly how the learning process uses train datapoints to predict on the target tasks.
arXiv Detail & Related papers (2024-01-23T17:22:00Z) - Provably Efficient Offline Reinforcement Learning with Perturbed Data
Sources [23.000116974718]
Existing theoretical studies on offline reinforcement learning (RL) mostly consider a dataset sampled directly from the target task.
In practice, however, data often come from several heterogeneous but related sources.
This work aims at rigorously understanding offline RL with multiple datasets that are collected from randomly perturbed versions of the target task instead of from itself.
arXiv Detail & Related papers (2023-06-14T08:53:20Z) - To Share or not to Share: Predicting Sets of Sources for Model Transfer
Learning [22.846469609263416]
We study the effects of model transfer on sequence labeling across various domains and tasks.
Our methods based on model similarity and support vector machines are able to predict promising sources, resulting in performance increases of up to 24 F1 points.
arXiv Detail & Related papers (2021-04-16T12:44:40Z) - Unsupervised Multi-source Domain Adaptation Without Access to Source
Data [58.551861130011886]
Unsupervised Domain Adaptation (UDA) aims to learn a predictor model for an unlabeled domain by transferring knowledge from a separate labeled source domain.
We propose a novel and efficient algorithm which automatically combines the source models with suitable weights in such a way that it performs at least as good as the best source model.
arXiv Detail & Related papers (2021-04-05T10:45:12Z) - Domain Impression: A Source Data Free Domain Adaptation Method [27.19677042654432]
Unsupervised domain adaptation methods solve the adaptation problem for an unlabeled target set, assuming that the source dataset is available with all labels.
This paper proposes a domain adaptation technique that does not need any source data.
Instead of the source data, we are only provided with a classifier that is trained on the source data.
arXiv Detail & Related papers (2021-02-17T19:50:49Z) - Multi-characteristic Subject Selection from Biased Datasets [79.82881947891589]
We present a constrained optimization-based method that finds the best possible sampling fractions for the different population subgroups.
Our results show that our proposed method outperforms the baselines for all problem variations by up to 90%.
arXiv Detail & Related papers (2020-12-18T15:55:27Z) - Reinforced Data Sampling for Model Diversification [15.547681142342846]
This paper proposes a new Reinforced Data Sampling (RDS) method to learn how to sample data adequately.
We formulate the optimisation problem of model diversification $delta-div$ in data sampling to maximise learning potentials and optimum allocation by injecting model diversity.
Our results suggest that the trainable sampling for model diversification is useful for competition organisers, researchers, or even starters to pursue full potentials of various machine learning tasks.
arXiv Detail & Related papers (2020-06-12T11:46:13Z) - Do We Really Need to Access the Source Data? Source Hypothesis Transfer
for Unsupervised Domain Adaptation [102.67010690592011]
Unsupervised adaptationUDA (UDA) aims to leverage the knowledge learned from a labeled source dataset to solve similar tasks in a new unlabeled domain.
Prior UDA methods typically require to access the source data when learning to adapt the model.
This work tackles a practical setting where only a trained source model is available and how we can effectively utilize such a model without source data to solve UDA problems.
arXiv Detail & Related papers (2020-02-20T03:13:58Z) - Improving Multi-Turn Response Selection Models with Complementary
Last-Utterance Selection by Instance Weighting [84.9716460244444]
We consider utilizing the underlying correlation in the data resource itself to derive different kinds of supervision signals.
We conduct extensive experiments in two public datasets and obtain significant improvement in both datasets.
arXiv Detail & Related papers (2020-02-18T06:29:01Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.