IMO: Greedy Layer-Wise Sparse Representation Learning for Out-of-Distribution Text Classification with Pre-trained Models

• URL: http://arxiv.org/abs/2404.13504v1
• Date: Sun, 21 Apr 2024 02:15:59 GMT
• Title: IMO: Greedy Layer-Wise Sparse Representation Learning for Out-of-Distribution Text Classification with Pre-trained Models
• Authors: Tao Feng, Lizhen Qu, Zhuang Li, Haolan Zhan, Yuncheng Hua, Gholamreza Haffari,
• Abstract summary: This study focuses on a specific problem of domain generalization, where a model is trained on one source domain and tested on multiple target domains that are unseen during training. We propose IMO: Invariant features Masks for Out-of-Distribution text classification, to achieve OOD generalization by learning invariant features.
• Score: 56.10157988449818
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Machine learning models have made incredible progress, but they still struggle when applied to examples from unseen domains. This study focuses on a specific problem of domain generalization, where a model is trained on one source domain and tested on multiple target domains that are unseen during training. We propose IMO: Invariant features Masks for Out-of-Distribution text classification, to achieve OOD generalization by learning invariant features. During training, IMO would learn sparse mask layers to remove irrelevant features for prediction, where the remaining features keep invariant. Additionally, IMO has an attention module at the token level to focus on tokens that are useful for prediction. Our comprehensive experiments show that IMO substantially outperforms strong baselines in terms of various evaluation metrics and settings.

Related papers

• Language Models for Text Classification: Is In-Context Learning Enough? [54.869097980761595]
  Recent foundational language models have shown state-of-the-art performance in many NLP tasks in zero- and few-shot settings. An advantage of these models over more standard approaches is the ability to understand instructions written in natural language (prompts) This makes them suitable for addressing text classification problems for domains with limited amounts of annotated instances.
  arXiv  Detail & Related papers  (2024-03-26T12:47:39Z)
• Multivariate Prototype Representation for Domain-Generalized Incremental Learning [35.83706574551515]
  We design a DGCIL approach that remembers old classes, adapts to new classes, and can classify reliably objects from unseen domains. Our loss formulation maintains classification boundaries and suppresses the domain-specific information of each class.
  arXiv  Detail & Related papers  (2023-09-24T06:42:04Z)
• Self-Evolution Learning for Discriminative Language Model Pretraining [103.57103957631067]
  Self-Evolution learning (SE) is a simple and effective token masking and learning method. SE focuses on learning the informative yet under-explored tokens and adaptively regularizes the training by introducing a novel Token-specific Label Smoothing approach.
  arXiv  Detail & Related papers  (2023-05-24T16:00:54Z)
• Distributional Shift Adaptation using Domain-Specific Features [41.91388601229745]
  In open-world scenarios, streaming big data can be Out-Of-Distribution (OOD) We propose a simple yet effective approach that relies on correlations in general regardless of whether the features are invariant or not. Our approach uses the most confidently predicted samples identified by an OOD base model to train a new model that effectively adapts to the target domain.
  arXiv  Detail & Related papers  (2022-11-09T04:16:21Z)
• An Additive Instance-Wise Approach to Multi-class Model Interpretation [53.87578024052922]
  Interpretable machine learning offers insights into what factors drive a certain prediction of a black-box system. Existing methods mainly focus on selecting explanatory input features, which follow either locally additive or instance-wise approaches. This work exploits the strengths of both methods and proposes a global framework for learning local explanations simultaneously for multiple target classes.
  arXiv  Detail & Related papers  (2022-07-07T06:50:27Z)
• Few-Shot Classification in Unseen Domains by Episodic Meta-Learning Across Visual Domains [36.98387822136687]
  Few-shot classification aims to carry out classification given only few labeled examples for the categories of interest. In this paper, we present a unique learning framework for domain-generalized few-shot classification. By advancing meta-learning strategies, our learning framework exploits data across multiple source domains to capture domain-invariant features.
  arXiv  Detail & Related papers  (2021-12-27T06:54:11Z)
• Learning to Ignore: Fair and Task Independent Representations [0.7106986689736827]
  In this work we show that they can be seen as a common framework of learning invariant representations. The representations should allow to predict the target while at the same time being invariant to sensitive attributes which split the dataset into subgroups. Our approach is based on the simple observation that it is impossible for any learning algorithm to differentiate samples if they have the same feature representation.
  arXiv  Detail & Related papers  (2021-01-11T17:33:18Z)
• Adaptive Risk Minimization: Learning to Adapt to Domain Shift [109.87561509436016]
  A fundamental assumption of most machine learning algorithms is that the training and test data are drawn from the same underlying distribution. In this work, we consider the problem setting of domain generalization, where the training data are structured into domains and there may be multiple test time shifts. We introduce the framework of adaptive risk minimization (ARM), in which models are directly optimized for effective adaptation to shift by learning to adapt on the training domains.
  arXiv  Detail & Related papers  (2020-07-06T17:59:30Z)
• Few-Shot Learning as Domain Adaptation: Algorithm and Analysis [120.75020271706978]
  Few-shot learning uses prior knowledge learned from the seen classes to recognize the unseen classes. This class-difference-caused distribution shift can be considered as a special case of domain shift. We propose a prototypical domain adaptation network with attention (DAPNA) to explicitly tackle such a domain shift problem in a meta-learning framework.
  arXiv  Detail & Related papers  (2020-02-06T01:04:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.