Regex-augmented Domain Transfer Topic Classification based on a Pre-trained Language Model: An application in Financial Domain

• URL: http://arxiv.org/abs/2305.18324v1
• Date: Tue, 23 May 2023 03:26:32 GMT
• Title: Regex-augmented Domain Transfer Topic Classification based on a Pre-trained Language Model: An application in Financial Domain
• Authors: Vanessa Liao, Syed Shariyar Murtaza, Yifan Nie, Jimmy Lin
• Abstract summary: We discuss the use of regular expression patterns employed as features for domain knowledge during the process of fine tuning. Our experiments on real scenario production data show that this method of fine tuning improves the downstream text classification tasks.
• Score: 42.5087655999509
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: A common way to use large pre-trained language models for downstream tasks is to fine tune them using additional layers. This may not work well if downstream domain is a specialized domain whereas the large language model has been pre-trained on a generic corpus. In this paper, we discuss the use of regular expression patterns employed as features for domain knowledge during the process of fine tuning, in addition to domain specific text. Our experiments on real scenario production data show that this method of fine tuning improves the downstream text classification tasks as compared to fine tuning only on domain specific text. We also show that the use of attention network for fine tuning improves results compared to simple linear layers.

Related papers

• Improving Domain Adaptation through Extended-Text Reading Comprehension [108.24709810480654]
  Recent work demonstrates that models using reading comprehension data formatted by adapting-based patterns can significantly improve performance on domain-specific tasks. However, these patterns are incapable of parsing raw corpora using domain-specific knowledge. In comparison to AdaptLLM, our method achieves an improvement exceeding 5% in domain-specific tasks.
  arXiv  Detail & Related papers  (2024-01-14T13:11:31Z)
• SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domains [14.096170976149521]
  SwitchPrompt is a novel and lightweight prompting methodology for adaptation of language models trained on datasets from the general domain to diverse low-resource domains. Our few-shot experiments on three text classification benchmarks demonstrate the efficacy of the general-domain pre-trained language models when used with SwitchPrompt. They often even outperform their domain-specific counterparts trained with baseline state-of-the-art prompting methods by up to 10.7% performance increase in accuracy.
  arXiv  Detail & Related papers  (2023-02-14T07:14:08Z)
• General-to-Specific Transfer Labeling for Domain Adaptable Keyphrase Generation [30.167332489528608]
  Training keyphrase generation (KPG) models require a large amount of annotated data, which can be prohibitively expensive and often limited to specific domains. We propose a three-stage pipeline, which gradually guides KPG models' learning focus from general syntactical features to domain-related semantics. Experiment results show that the proposed process can produce good-quality keyphrases in new domains and achieve consistent improvements after adaptation with limited in-domain annotated data.
  arXiv  Detail & Related papers  (2022-08-20T04:43:01Z)
• Task Transfer and Domain Adaptation for Zero-Shot Question Answering [18.188082154309175]
  We use supervised pretraining on source-domain data to reduce sample complexity on domain-specific downstream tasks. We evaluate zero-shot performance on domain-specific reading comprehension tasks by combining task transfer with domain adaptation.
  arXiv  Detail & Related papers  (2022-06-14T09:10:48Z)
• Efficient Hierarchical Domain Adaptation for Pretrained Language Models [77.02962815423658]
  Generative language models are trained on diverse, general domain corpora. We introduce a method to scale domain adaptation to many diverse domains using a computationally efficient adapter approach.
  arXiv  Detail & Related papers  (2021-12-16T11:09:29Z)
• Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains [45.07506437436464]
  We present a general approach to developing small, fast and effective pre-trained models for specific domains. This is achieved by adapting the off-the-shelf general pre-trained models and performing task-agnostic knowledge distillation in target domains.
  arXiv  Detail & Related papers  (2021-06-25T07:37:05Z)
• Batch Normalization Embeddings for Deep Domain Generalization [50.51405390150066]
  Domain generalization aims at training machine learning models to perform robustly across different and unseen domains. We show a significant increase in classification accuracy over current state-of-the-art techniques on popular domain generalization benchmarks.
  arXiv  Detail & Related papers  (2020-11-25T12:02:57Z)
• Domain Adversarial Fine-Tuning as an Effective Regularizer [80.14528207465412]
  In Natural Language Processing (NLP), pretrained language models (LMs) that are transferred to downstream tasks have been recently shown to achieve state-of-the-art results. Standard fine-tuning can degrade the general-domain representations captured during pretraining. We introduce a new regularization technique, AFTER; domain Adversarial Fine-Tuning as an Effective Regularizer.
  arXiv  Detail & Related papers  (2020-09-28T14:35:06Z)
• Unsupervised Domain Clusters in Pretrained Language Models [61.832234606157286]
  We show that massive pre-trained language models implicitly learn sentence representations that cluster by domains without supervision. We propose domain data selection methods based on such models. We evaluate our data selection methods for neural machine translation across five diverse domains.
  arXiv  Detail & Related papers  (2020-04-05T06:22:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.