Improving Domain Adaptation through Extended-Text Reading Comprehension

• URL: http://arxiv.org/abs/2401.07284v2
• Date: Thu, 18 Jan 2024 11:29:37 GMT
• Title: Improving Domain Adaptation through Extended-Text Reading Comprehension
• Authors: Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang
• Abstract summary: Recent work demonstrates that models using reading comprehension data formatted by adapting-based patterns can significantly improve performance on domain-specific tasks. However, these patterns are incapable of parsing raw corpora using domain-specific knowledge. In comparison to AdaptLLM, our method achieves an improvement exceeding 5% in domain-specific tasks.
• Score: 108.24709810480654
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To enhance the domain-specific capabilities of large language models, continued pre-training on a domain-specific corpus is a prevalent method. Recent work demonstrates that adapting models using reading comprehension data formatted by regex-based patterns can significantly improve performance on domain-specific tasks. However, regex-based patterns are incapable of parsing raw corpora using domain-specific knowledge. Furthermore, the question and answer pairs are extracted directly from the corpus in predefined formats offers limited context. To address this limitation, we improve reading comprehension via LLM and clustering. LLM focuses on leveraging domain knowledge within the corpus to refine comprehension stage, while clustering supplies relevant knowledge by extending the context to enrich reading stage. Additionally, our method incorporates parameter-efficient fine-tuning to improve the efficiency of domain adaptation. In comparison to AdaptLLM, our method achieves an improvement exceeding 5% in domain-specific tasks. Our code will available at https://github.com/microsoft/LMOps.

Related papers

• AdaptEval: Evaluating Large Language Models on Domain Adaptation for Text Summarization [4.07484910093752]
  We evaluate the domain adaptation abilities of Large Language Models (LLM) on the summarization task across various domains. We present AdaptEval, the first domain adaptation evaluation suite.
  arXiv  Detail & Related papers  (2024-07-16T10:50:39Z)
• Cross-domain Chinese Sentence Pattern Parsing [67.1381983012038]
  Sentence Pattern Structure (SPS) parsing is a syntactic analysis method primarily employed in language teaching. Existing SPSs rely heavily on textbook corpora for training, lacking cross-domain capability. This paper proposes an innovative approach leveraging large language models (LLMs) within a self-training framework.
  arXiv  Detail & Related papers  (2024-02-26T05:30:48Z)
• Leveraging Code to Improve In-context Learning for Semantic Parsing [48.66031267718704]
  In-context learning (ICL) is an appealing approach for semantic parsing due to its few-shot nature and improved generalization. We improve the effectiveness of ICL for semantic parsing by (1) using general-purpose programming languages such as Python instead of DSLs, and (2) augmenting prompts with a structured domain description.
  arXiv  Detail & Related papers  (2023-11-16T02:50:06Z)
• Adapting Large Language Models to Domains via Reading Comprehension [86.24451681746676]
  We explore how continued pre-training on domain-specific corpora influences large language models. We show that training on the raw corpora endows the model with domain knowledge, but drastically hurts its ability for question answering. We propose a simple method for transforming raw corpora into reading comprehension texts.
  arXiv  Detail & Related papers  (2023-09-18T07:17:52Z)
• Regex-augmented Domain Transfer Topic Classification based on a Pre-trained Language Model: An application in Financial Domain [42.5087655999509]
  We discuss the use of regular expression patterns employed as features for domain knowledge during the process of fine tuning. Our experiments on real scenario production data show that this method of fine tuning improves the downstream text classification tasks.
  arXiv  Detail & Related papers  (2023-05-23T03:26:32Z)
• VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding [5.1282202633907]
  We propose a novel Transformer-based language model named VarMAE for domain-adaptive language understanding. Under the masked autoencoding objective, we design a context uncertainty learning module to encode the token's context into a smooth latent distribution. Experiments on science- and finance-domain NLU tasks demonstrate that VarMAE can be efficiently adapted to new domains with limited resources.
  arXiv  Detail & Related papers  (2022-11-01T12:51:51Z)
• Meta-Learning for Domain Generalization in Semantic Parsing [124.32975734073949]
  We use a meta-learning framework which targets zero-shot domain for semantic parsing. We apply a model-agnostic training algorithm that simulates zero-shot parsing virtual train and test sets from disjoint domains.
  arXiv  Detail & Related papers  (2020-10-22T19:00:36Z)
• Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing [85.35582118010608]
  Task-oriented semantic parsing is a critical component of virtual assistants. Recent advances in deep learning have enabled several approaches to successfully parse more complex queries. We propose a novel method that outperforms a supervised neural model at a 10-fold data reduction.
  arXiv  Detail & Related papers  (2020-10-07T17:47:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.