Boosting Binary Masks for Multi-Domain Learning through Affine Transformations

• URL: http://arxiv.org/abs/2103.13894v1
• Date: Thu, 25 Mar 2021 14:54:37 GMT
• Title: Boosting Binary Masks for Multi-Domain Learning through Affine Transformations
• Authors: Massimiliano Mancini, Elisa Ricci, Barbara Caputo and Samuel Rota Bul\'o
• Abstract summary: The goal of multi-domain learning is to produce a single model performing a task in all the domains together. Recent works showed how we can address this problem by masking the internal weights of a given original conv-net through learned binary variables. We provide a general formulation of binary mask based models for multi-domain learning by affine transformations of the original network parameters.
• Score: 49.25451497933657
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we present a new, algorithm for multi-domain learning. Given a pretrained architecture and a set of visual domains received sequentially, the goal of multi-domain learning is to produce a single model performing a task in all the domains together. Recent works showed how we can address this problem by masking the internal weights of a given original conv-net through learned binary variables. In this work, we provide a general formulation of binary mask based models for multi-domain learning by affine transformations of the original network parameters. Our formulation obtains significantly higher levels of adaptation to new domains, achieving performances comparable to domain-specific models while requiring slightly more than 1 bit per network parameter per additional domain. Experiments on two popular benchmarks showcase the power of our approach, achieving performances close to state-of-the-art methods on the Visual Decathlon Challenge.

Related papers

• Investigating the potential of Sparse Mixtures-of-Experts for multi-domain neural machine translation [59.41178047749177]
  We focus on multi-domain Neural Machine Translation, with the goal of developing efficient models which can handle data from various domains seen during training and are robust to domains unseen during training. We hypothesize that Sparse Mixture-of-Experts (SMoE) models are a good fit for this task, as they enable efficient model scaling. We conduct a series of experiments aimed at validating the utility of SMoE for the multi-domain scenario, and find that a straightforward width scaling of Transformer is a simpler and surprisingly more efficient approach in practice, and reaches the same performance level as SMoE.
  arXiv  Detail & Related papers  (2024-07-01T09:45:22Z)
• Multi-Domain Learning with Modulation Adapters [33.54630534228469]
  Multi-domain learning aims to handle related tasks, such as image classification across multiple domains, simultaneously. Modulation Adapters update the convolutional weights of the model in a multiplicative manner for each task. Our approach yields excellent results, with accuracies that are comparable to or better than those of existing state-of-the-art approaches.
  arXiv  Detail & Related papers  (2023-07-17T14:40:16Z)
• Exploring Distributional Shifts in Large Language Models for Code Analysis [36.73114441988879]
  We study how three large language models with code capabilities generalize to out-of-domain data. We consider two fundamental applications - code summarization, and code generation. We find that a model adapted to multiple domains simultaneously performs on par with those adapted to a single domain.
  arXiv  Detail & Related papers  (2023-03-16T07:45:46Z)
• Learning Domain Invariant Prompt for Vision-Language Models [31.581652862478965]
  We propose a novel prompt learning paradigm that directly generates emphdomain invariant prompt that can be generalized to unseen domains, called MetaPrompt. Our method consistently and significantly outperforms existing methods.
  arXiv  Detail & Related papers  (2022-12-08T11:23:24Z)
• Domain Invariant Masked Autoencoders for Self-supervised Learning from Multi-domains [73.54897096088149]
  We propose a Domain-invariant Masked AutoEncoder (DiMAE) for self-supervised learning from multi-domains. The core idea is to augment the input image with style noise from different domains and then reconstruct the image from the embedding of the augmented image. Experiments on PACS and DomainNet illustrate that DiMAE achieves considerable gains compared with recent state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-05-10T09:49:40Z)
• Efficient Hierarchical Domain Adaptation for Pretrained Language Models [77.02962815423658]
  Generative language models are trained on diverse, general domain corpora. We introduce a method to scale domain adaptation to many diverse domains using a computationally efficient adapter approach.
  arXiv  Detail & Related papers  (2021-12-16T11:09:29Z)
• TAL: Two-stream Adaptive Learning for Generalizable Person Re-identification [115.31432027711202]
  We argue that both domain-specific and domain-invariant features are crucial for improving the generalization ability of re-id models. We name two-stream adaptive learning (TAL) to simultaneously model these two kinds of information. Our framework can be applied to both single-source and multi-source domain generalization tasks.
  arXiv  Detail & Related papers  (2021-11-29T01:27:42Z)
• Universal Representation Learning from Multiple Domains for Few-shot Classification [41.821234589075445]
  We propose to learn a single set of universal deep representations by distilling knowledge of multiple separately trained networks. We show that the universal representations can be further refined for previously unseen domains by an efficient adaptation step.
  arXiv  Detail & Related papers  (2021-03-25T13:49:12Z)
• Not all domains are equally complex: Adaptive Multi-Domain Learning [98.25886129591974]
  We propose an adaptive parameterization approach to deep neural networks for multi-domain learning. The proposed approach performs on par with the original approach while reducing by far the number of parameters.
  arXiv  Detail & Related papers  (2020-03-25T17:16:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.