Learning Neural Models for Natural Language Processing in the Face of
Distributional Shift
- URL: http://arxiv.org/abs/2109.01558v1
- Date: Fri, 3 Sep 2021 14:29:20 GMT
- Title: Learning Neural Models for Natural Language Processing in the Face of
Distributional Shift
- Authors: Paul Michel
- Abstract summary: The dominating NLP paradigm of training a strong neural predictor to perform one task on a specific dataset has led to state-of-the-art performance in a variety of applications.
It builds upon the assumption that the data distribution is stationary, ie. that the data is sampled from a fixed distribution both at training and test time.
This way of training is inconsistent with how we as humans are able to learn from and operate within a constantly changing stream of information.
It is ill-adapted to real-world use cases where the data distribution is expected to shift over the course of a model's lifetime
- Score: 10.990447273771592
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The dominating NLP paradigm of training a strong neural predictor to perform
one task on a specific dataset has led to state-of-the-art performance in a
variety of applications (eg. sentiment classification, span-prediction based
question answering or machine translation). However, it builds upon the
assumption that the data distribution is stationary, ie. that the data is
sampled from a fixed distribution both at training and test time. This way of
training is inconsistent with how we as humans are able to learn from and
operate within a constantly changing stream of information. Moreover, it is
ill-adapted to real-world use cases where the data distribution is expected to
shift over the course of a model's lifetime.
The first goal of this thesis is to characterize the different forms this
shift can take in the context of natural language processing, and propose
benchmarks and evaluation metrics to measure its effect on current deep
learning architectures. We then proceed to take steps to mitigate the effect of
distributional shift on NLP models. To this end, we develop methods based on
parametric reformulations of the distributionally robust optimization
framework. Empirically, we demonstrate that these approaches yield more robust
models as demonstrated on a selection of realistic problems. In the third and
final part of this thesis, we explore ways of efficiently adapting existing
models to new domains or tasks. Our contribution to this topic takes
inspiration from information geometry to derive a new gradient update rule
which alleviate catastrophic forgetting issues during adaptation.
Related papers
- Optimal Classification under Performative Distribution Shift [13.508249764979075]
We propose a novel view in which performative effects are modelled as push-forward measures.
We prove the convexity of the performative risk under a new set of assumptions.
We also establish a connection with adversarially robust classification by reformulating the minimization of the performative risk as a min-max variational problem.
arXiv Detail & Related papers (2024-11-04T12:20:13Z) - Adjusting Pretrained Backbones for Performativity [34.390793811659556]
We propose a novel technique to adjust pretrained backbones for performativity in a modular way.
We show how it leads to smaller loss along the retraining trajectory and enables us to effectively select among candidate models to anticipate performance degradations.
arXiv Detail & Related papers (2024-10-06T14:41:13Z) - Distilled Datamodel with Reverse Gradient Matching [74.75248610868685]
We introduce an efficient framework for assessing data impact, comprising offline training and online evaluation stages.
Our proposed method achieves comparable model behavior evaluation while significantly speeding up the process compared to the direct retraining method.
arXiv Detail & Related papers (2024-04-22T09:16:14Z) - Federated Learning with Projected Trajectory Regularization [65.6266768678291]
Federated learning enables joint training of machine learning models from distributed clients without sharing their local data.
One key challenge in federated learning is to handle non-identically distributed data across the clients.
We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
arXiv Detail & Related papers (2023-12-22T02:12:08Z) - Prior-guided Source-free Domain Adaptation for Human Pose Estimation [24.50953879583841]
Domain adaptation methods for 2D human pose estimation typically require continuous access to the source data.
We present Prior-guided Self-training (POST), a pseudo-labeling approach that builds on the popular Mean Teacher framework.
arXiv Detail & Related papers (2023-08-26T20:30:04Z) - Consistency Regularization for Generalizable Source-free Domain
Adaptation [62.654883736925456]
Source-free domain adaptation (SFDA) aims to adapt a well-trained source model to an unlabelled target domain without accessing the source dataset.
Existing SFDA methods ONLY assess their adapted models on the target training set, neglecting the data from unseen but identically distributed testing sets.
We propose a consistency regularization framework to develop a more generalizable SFDA method.
arXiv Detail & Related papers (2023-08-03T07:45:53Z) - On Generalizing Beyond Domains in Cross-Domain Continual Learning [91.56748415975683]
Deep neural networks often suffer from catastrophic forgetting of previously learned knowledge after learning a new task.
Our proposed approach learns new tasks under domain shift with accuracy boosts up to 10% on challenging datasets such as DomainNet and OfficeHome.
arXiv Detail & Related papers (2022-03-08T09:57:48Z) - Data Augmentation through Expert-guided Symmetry Detection to Improve
Performance in Offline Reinforcement Learning [0.0]
offline estimation of the dynamical model of a Markov Decision Process (MDP) is a non-trivial task.
Recent works showed that an expert-guided pipeline relying on Density Estimation methods effectively detects this structure in deterministic environments.
We show that the former results lead to a performance improvement when solving the learned MDP and then applying the optimized policy in the real environment.
arXiv Detail & Related papers (2021-12-18T14:32:32Z) - Post-Hoc Domain Adaptation via Guided Data Homogenization [0.0]
We propose to deal with changes in the data distribution via guided data homogenization.
This approach makes use of information about the training data contained implicitly in the deep learning model to learn a domain transfer function.
We demonstrate the potential of data homogenization through experiments on the CIFAR-10 and MNIST data sets.
arXiv Detail & Related papers (2021-04-08T09:18:48Z) - Exploring Complementary Strengths of Invariant and Equivariant
Representations for Few-Shot Learning [96.75889543560497]
In many real-world problems, collecting a large number of labeled samples is infeasible.
Few-shot learning is the dominant approach to address this issue, where the objective is to quickly adapt to novel categories in presence of a limited number of samples.
We propose a novel training mechanism that simultaneously enforces equivariance and invariance to a general set of geometric transformations.
arXiv Detail & Related papers (2021-03-01T21:14:33Z) - Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting.
Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking.
We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
arXiv Detail & Related papers (2020-10-24T11:55:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.