Example-based Hypernetworks for Out-of-Distribution Generalization

• URL: http://arxiv.org/abs/2203.14276v3
• Date: Wed, 18 Oct 2023 19:30:13 GMT
• Title: Example-based Hypernetworks for Out-of-Distribution Generalization
• Authors: Tomer Volk, Eyal Ben-David, Ohad Amosy, Gal Chechik, Roi Reichart
• Abstract summary: This paper addresses the issue of multi-source adaptation for unfamiliar domains. We leverage labeled data from multiple source domains to generalize to unknown target domains at training. Our innovative framework employs example-based Hypernetwork adaptation.
• Score: 43.99982893976026
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As Natural Language Processing (NLP) algorithms continually achieve new milestones, out-of-distribution generalization remains a significant challenge. This paper addresses the issue of multi-source adaptation for unfamiliar domains: We leverage labeled data from multiple source domains to generalize to unknown target domains at training. Our innovative framework employs example-based Hypernetwork adaptation: a T5 encoder-decoder initially generates a unique signature from an input example, embedding it within the source domains' semantic space. This signature is subsequently utilized by a Hypernetwork to generate the task classifier's weights. We evaluated our method across two tasks - sentiment classification and natural language inference - in 29 adaptation scenarios, where it outpaced established algorithms. In an advanced version, the signature also enriches the input example's representation. We also compare our finetuned architecture to few-shot GPT-3, demonstrating its effectiveness in essential use cases. To our knowledge, this marks the first application of Hypernetworks to the adaptation for unknown domains.

Related papers

• Generalized One-shot Domain Adaption of Generative Adversarial Networks [72.84435077616135]
  The adaption of Generative Adversarial Network (GAN) aims to transfer a pre-trained GAN to a given domain with limited training data. We consider that the adaptation from source domain to target domain can be decoupled into two parts: the transfer of global style like texture and color, and the emergence of new entities that do not belong to the source domain. Our core objective is to constrain the gap between the internal distributions of the reference and syntheses by sliced Wasserstein distance.
  arXiv  Detail & Related papers  (2022-09-08T09:24:44Z)
• Dynamic Instance Domain Adaptation [109.53575039217094]
  Most studies on unsupervised domain adaptation assume that each domain's training samples come with domain labels. We develop a dynamic neural network with adaptive convolutional kernels to generate instance-adaptive residuals to adapt domain-agnostic deep features to each individual instance. Our model, dubbed DIDA-Net, achieves state-of-the-art performance on several commonly used single-source and multi-source UDA datasets.
  arXiv  Detail & Related papers  (2022-03-09T20:05:54Z)
• Adaptive Methods for Aggregated Domain Generalization [26.215904177457997]
  In many settings, privacy concerns prohibit obtaining domain labels for the training data samples. We propose a domain-adaptive approach to this problem, which operates in two steps. Our approach achieves state-of-the-art performance on a variety of domain generalization benchmarks without using domain labels.
  arXiv  Detail & Related papers  (2021-12-09T08:57:01Z)
• Curriculum Graph Co-Teaching for Multi-Target Domain Adaptation [78.28390172958643]
  We identify two key aspects that can help to alleviate multiple domain-shifts in the multi-target domain adaptation (MTDA) We propose Curriculum Graph Co-Teaching (CGCT) that uses a dual classifier head, with one of them being a graph convolutional network (GCN) which aggregates features from similar samples across the domains. When the domain labels are available, we propose Domain-aware Curriculum Learning (DCL), a sequential adaptation strategy that first adapts on the easier target domains, followed by the harder ones.
  arXiv  Detail & Related papers  (2021-04-01T23:41:41Z)
• PADA: A Prompt-based Autoregressive Approach for Adaptation to Unseen Domains [19.682729518136142]
  PADA: A Prompt-based Autoregressive Domain Adaptation algorithm, based on the T5 model. We present PADA: A Prompt-based Autoregressive Domain Adaptation algorithm, based on the T5 model. In experiments with two tasks, PADA strongly outperforms state-of-the-art approaches and additional strong baselines.
  arXiv  Detail & Related papers  (2021-02-24T11:02:29Z)
• Robust Domain-Free Domain Generalization with Class-aware Alignment [4.442096198968069]
  Domain-Free Domain Generalization (DFDG) is a model-agnostic method to achieve better generalization performance on the unseen test domain. DFDG uses novel strategies to learn domain-invariant class-discriminative features. It obtains competitive performance on both time series sensor and image classification public datasets.
  arXiv  Detail & Related papers  (2021-02-17T17:46:06Z)
• Curriculum CycleGAN for Textual Sentiment Domain Adaptation with Multiple Sources [68.31273535702256]
  We propose a novel instance-level MDA framework, named curriculum cycle-consistent generative adversarial network (C-CycleGAN) C-CycleGAN consists of three components: (1) pre-trained text encoder which encodes textual input from different domains into a continuous representation space, (2) intermediate domain generator with curriculum instance-level adaptation which bridges the gap across source and target domains, and (3) task classifier trained on the intermediate domain for final sentiment classification. We conduct extensive experiments on three benchmark datasets and achieve substantial gains over state-of-the-art DA approaches.
  arXiv  Detail & Related papers  (2020-11-17T14:50:55Z)
• Supervised Domain Adaptation using Graph Embedding [86.3361797111839]
  Domain adaptation methods assume that distributions between the two domains are shifted and attempt to realign them. We propose a generic framework based on graph embedding. We show that the proposed approach leads to a powerful Domain Adaptation framework.
  arXiv  Detail & Related papers  (2020-03-09T12:25:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.