Multi-BERT: Leveraging Adapters and Prompt Tuning for Low-Resource Multi-Domain Adaptation

• URL: http://arxiv.org/abs/2404.02335v1
• Date: Tue, 2 Apr 2024 22:15:48 GMT
• Title: Multi-BERT: Leveraging Adapters and Prompt Tuning for Low-Resource Multi-Domain Adaptation
• Authors: Parham Abed Azad, Hamid Beigy,
• Abstract summary: The rapid expansion of texts' volume and diversity presents formidable challenges in multi-domain settings. Traditional approaches, either employing a unified model for multiple domains or individual models for each domain, frequently pose significant limitations. This paper introduces a novel approach composed of one core model with multiple sets of domain-specific parameters.
• Score: 14.211024633768986
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapid expansion of texts' volume and diversity presents formidable challenges in multi-domain settings. These challenges are also visible in the Persian name entity recognition (NER) settings. Traditional approaches, either employing a unified model for multiple domains or individual models for each domain, frequently pose significant limitations. Single models often struggle to capture the nuances of diverse domains, while utilizing multiple large models can lead to resource constraints, rendering the training of a model for each domain virtually impractical. Therefore, this paper introduces a novel approach composed of one core model with multiple sets of domain-specific parameters. We utilize techniques such as prompt tuning and adapters, combined with the incorporation of additional layers, to add parameters that we can train for the specific domains. This enables the model to perform comparably to individual models for each domain. Experimental results on different formal and informal datasets show that by employing these added parameters, the proposed model significantly surpasses existing practical models in performance. Remarkably, the proposed model requires only one instance for training and storage, yet achieves outstanding results across all domains, even surpassing the state-of-the-art in some. Moreover, we analyze each adaptation strategy, delineating its strengths, weaknesses, and optimal hyper-parameters for the Persian NER settings. Finally, we introduce a document-based domain detection pipeline tailored for scenarios with unknown text domains, enhancing the adaptability and practicality of this paper in real-world applications.

Related papers

• FDS: Feedback-guided Domain Synthesis with Multi-Source Conditional Diffusion Models for Domain Generalization [19.0284321951354]
  Domain Generalization techniques aim to enhance model robustness by simulating novel data distributions during training. We propose FDS, Feedback-guided Domain Synthesis, a novel strategy that employs diffusion models to synthesize novel, pseudo-domains. Our evaluations demonstrate that this methodology sets new benchmarks in domain generalization performance across a range of challenging datasets.
  arXiv  Detail & Related papers  (2024-07-04T02:45:29Z)
• GM-DF: Generalized Multi-Scenario Deepfake Detection [49.072106087564144]
  Existing face forgery detection usually follows the paradigm of training models in a single domain. In this paper, we elaborately investigate the generalization capacity of deepfake detection models when jointly trained on multiple face forgery detection datasets.
  arXiv  Detail & Related papers  (2024-06-28T17:42:08Z)
• Unified Language-driven Zero-shot Domain Adaptation [55.64088594551629]
  Unified Language-driven Zero-shot Domain Adaptation (ULDA) is a novel task setting. It enables a single model to adapt to diverse target domains without explicit domain-ID knowledge.
  arXiv  Detail & Related papers  (2024-04-10T16:44:11Z)
• Multi-Domain Learning with Modulation Adapters [33.54630534228469]
  Multi-domain learning aims to handle related tasks, such as image classification across multiple domains, simultaneously. Modulation Adapters update the convolutional weights of the model in a multiplicative manner for each task. Our approach yields excellent results, with accuracies that are comparable to or better than those of existing state-of-the-art approaches.
  arXiv  Detail & Related papers  (2023-07-17T14:40:16Z)
• Exploring Distributional Shifts in Large Language Models for Code Analysis [36.73114441988879]
  We study how three large language models with code capabilities generalize to out-of-domain data. We consider two fundamental applications - code summarization, and code generation. We find that a model adapted to multiple domains simultaneously performs on par with those adapted to a single domain.
  arXiv  Detail & Related papers  (2023-03-16T07:45:46Z)
• AdaptDHM: Adaptive Distribution Hierarchical Model for Multi-Domain CTR Prediction [4.299153274884263]
  We propose an elegant and flexible multi-distribution modeling paradigm, named Adaptive Distribution Hierarchical Model (AdaptDHM) Our model achieves impressive prediction accuracy and its time cost during the training stage is more than 50% less than that of other models.
  arXiv  Detail & Related papers  (2022-11-22T09:10:37Z)
• Budget-Aware Pruning for Multi-Domain Learning [45.84899283894373]
  This work aims to prune models capable of handling multiple domains according to a user defined budget. We achieve this by encouraging all domains to use a similar subset of filters from the baseline model. The proposed approach innovates by better adapting to resource-limited devices.
  arXiv  Detail & Related papers  (2022-10-14T20:48:12Z)
• A Novel Mix-normalization Method for Generalizable Multi-source Person Re-identification [49.548815417844786]
  Person re-identification (Re-ID) has achieved great success in the supervised scenario. It is difficult to directly transfer the supervised model to arbitrary unseen domains due to the model overfitting to the seen source domains. We propose MixNorm, which consists of domain-aware mix-normalization (DMN) and domain-ware center regularization (DCR)
  arXiv  Detail & Related papers  (2022-01-24T18:09:38Z)
• TAL: Two-stream Adaptive Learning for Generalizable Person Re-identification [115.31432027711202]
  We argue that both domain-specific and domain-invariant features are crucial for improving the generalization ability of re-id models. We name two-stream adaptive learning (TAL) to simultaneously model these two kinds of information. Our framework can be applied to both single-source and multi-source domain generalization tasks.
  arXiv  Detail & Related papers  (2021-11-29T01:27:42Z)
• Multi-path Neural Networks for On-device Multi-domain Visual Classification [55.281139434736254]
  This paper proposes a novel approach to automatically learn a multi-path network for multi-domain visual classification on mobile devices. The proposed multi-path network is learned from neural architecture search by applying one reinforcement learning controller for each domain to select the best path in the super-network created from a MobileNetV3-like search space. The determined multi-path model selectively shares parameters across domains in shared nodes while keeping domain-specific parameters within non-shared nodes in individual domain paths.
  arXiv  Detail & Related papers  (2020-10-10T05:13:49Z)
• Not all domains are equally complex: Adaptive Multi-Domain Learning [98.25886129591974]
  We propose an adaptive parameterization approach to deep neural networks for multi-domain learning. The proposed approach performs on par with the original approach while reducing by far the number of parameters.
  arXiv  Detail & Related papers  (2020-03-25T17:16:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.