Related papers: Gated Domain Units for Multi-source Domain Generalization

Gated Domain Units for Multi-source Domain Generalization

URL: http://arxiv.org/abs/2206.12444v2
Date: Tue, 16 May 2023 14:32:42 GMT
Title: Gated Domain Units for Multi-source Domain Generalization
Authors: Simon F\"oll, Alina Dubatovka, Eugen Ernst, Siu Lun Chau, Martin Maritsch, Patrik Okanovic, Gudrun Th\"ater, Joachim M. Buhmann, Felix Wortmann, Krikamol Muandet
Abstract summary: Distribution shift (DS) occurs when a dataset at test time differs from the dataset at training time. We introduce a modular neural network layer consisting of Gated Domain Units (GDUs) that learn a representation for each latent elementary distribution. During inference, a weighted ensemble of learning machines can be created by comparing new observations with the representations of each elementary distribution.
Score: 14.643490853965385
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: The phenomenon of distribution shift (DS) occurs when a dataset at test time differs from the dataset at training time, which can significantly impair the performance of a machine learning model in practical settings due to a lack of knowledge about the data's distribution at test time. To address this problem, we postulate that real-world distributions are composed of latent Invariant Elementary Distributions (I.E.D) across different domains. This assumption implies an invariant structure in the solution space that enables knowledge transfer to unseen domains. To exploit this property for domain generalization, we introduce a modular neural network layer consisting of Gated Domain Units (GDUs) that learn a representation for each latent elementary distribution. During inference, a weighted ensemble of learning machines can be created by comparing new observations with the representations of each elementary distribution. Our flexible framework also accommodates scenarios where explicit domain information is not present. Extensive experiments on image, text, and graph data show consistent performance improvement on out-of-training target domains. These findings support the practicality of the I.E.D assumption and the effectiveness of GDUs for domain generalisation.