QUERT: Continual Pre-training of Language Model for Query Understanding in Travel Domain Search

• URL: http://arxiv.org/abs/2306.06707v1
• Date: Sun, 11 Jun 2023 15:39:59 GMT
• Title: QUERT: Continual Pre-training of Language Model for Query Understanding in Travel Domain Search
• Authors: Jian Xie, Yidan Liang, Jingping Liu, Yanghua Xiao, Baohua Wu, Shenghua Ni
• Abstract summary: We propose QUERT, A Continual Pre-trained Language Model for QUERy Understanding in Travel Domain Search. Quert is jointly trained on four tailored pre-training tasks to the characteristics of query in travel domain search. To check on the improvement of QUERT to online business, we deploy QUERT and perform A/B testing on Fliggy APP.
• Score: 15.026682829320261
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In light of the success of the pre-trained language models (PLMs), continual pre-training of generic PLMs has been the paradigm of domain adaption. In this paper, we propose QUERT, A Continual Pre-trained Language Model for QUERy Understanding in Travel Domain Search. QUERT is jointly trained on four tailored pre-training tasks to the characteristics of query in travel domain search: Geography-aware Mask Prediction, Geohash Code Prediction, User Click Behavior Learning, and Phrase and Token Order Prediction. Performance improvement of downstream tasks and ablation experiment demonstrate the effectiveness of our proposed pre-training tasks. To be specific, the average performance of downstream tasks increases by 2.02% and 30.93% in supervised and unsupervised settings, respectively. To check on the improvement of QUERT to online business, we deploy QUERT and perform A/B testing on Fliggy APP. The feedback results show that QUERT increases the Unique Click-Through Rate and Page Click-Through Rate by 0.89% and 1.03% when applying QUERT as the encoder. Our code and downstream task data will be released for future research.

Related papers

• AfroXLMR-Social: Adapting Pre-trained Language Models for African Languages Social Media Text [5.137881481160781]
  Pretrained Language Models (PLMs) built from various sources are the foundation of today's NLP progress. We explore a thorough analysis of domain and task adaptive continual pretraining approaches for low-resource African languages. We create AfriSocial, a corpus designed for domain adaptive finetuning that passes through quality pre-processing steps.
  arXiv  Detail & Related papers  (2025-03-24T00:06:33Z)
• Learn from the Learnt: Source-Free Active Domain Adaptation via Contrastive Sampling and Visual Persistence [60.37934652213881]
  Domain Adaptation (DA) facilitates knowledge transfer from a source domain to a related target domain. This paper investigates a practical DA paradigm, namely Source data-Free Active Domain Adaptation (SFADA), where source data becomes inaccessible during adaptation. We present learn from the learnt (LFTL), a novel paradigm for SFADA to leverage the learnt knowledge from the source pretrained model and actively iterated models without extra overhead.
  arXiv  Detail & Related papers  (2024-07-26T17:51:58Z)
• Value Explicit Pretraining for Learning Transferable Representations [11.069853883599102]
  We propose a method that learns generalizable representations for transfer reinforcement learning. We learn new tasks that share similar objectives as previously learned tasks, by learning an encoder for objective-conditioned representations. Experiments using a realistic navigation simulator and Atari benchmark show that the pretrained encoder produced by our method outperforms current SoTA pretraining methods.
  arXiv  Detail & Related papers  (2023-12-19T17:12:35Z)
• Self-supervised Adaptive Pre-training of Multilingual Speech Models for Language and Dialect Identification [19.893213508284813]
  Self-supervised adaptive pre-training is proposed to adapt the pre-trained model to the target domain and languages of the downstream task. We show that SAPT improves XLSR performance on the FLEURS benchmark with substantial gains up to 40.1% for under-represented languages.
  arXiv  Detail & Related papers  (2023-12-12T14:58:08Z)
• Masked Path Modeling for Vision-and-Language Navigation [41.7517631477082]
  Vision-and-language navigation (VLN) agents are trained to navigate in real-world environments by following natural language instructions. Previous approaches have attempted to address this issue by introducing additional supervision during training. We introduce a masked path modeling (MPM) objective, which pretrains an agent using self-collected data for downstream navigation tasks.
  arXiv  Detail & Related papers  (2023-05-23T17:20:20Z)
• ASPEST: Bridging the Gap Between Active Learning and Selective Prediction [56.001808843574395]
  Selective prediction aims to learn a reliable model that abstains from making predictions when uncertain. Active learning aims to lower the overall labeling effort, and hence human dependence, by querying the most informative examples. In this work, we introduce a new learning paradigm, active selective prediction, which aims to query more informative samples from the shifted target domain.
  arXiv  Detail & Related papers  (2023-04-07T23:51:07Z)
• Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Training [55.43088293183165]
  Recent studies show that pre-trained language models (PLMs) like BERT contain matchingworks that have similar transfer learning performance as the original PLM. In this paper, we find that the BERTworks have even more potential than these studies have shown. We train binary masks over model weights on the pre-training tasks, with the aim of preserving the universal transferability of the subnetwork.
  arXiv  Detail & Related papers  (2022-04-24T08:42:47Z)
• ELLE: Efficient Lifelong Pre-training for Emerging Data [91.52652408402815]
  Current pre-trained language models (PLM) are typically trained with static data, ignoring that in real-world scenarios, streaming data of various sources may continuously grow. We propose ELLE, aiming at efficient lifelong pre-training for emerging data. ELLE consists of (1) function preserved model expansion, which flexibly expands an existing PLM's width and depth to improve the efficiency of knowledge acquisition; and (2) pre-trained domain prompts, which disentangle the versatile knowledge learned during pre-training and stimulate the proper knowledge for downstream tasks.
  arXiv  Detail & Related papers  (2022-03-12T01:53:53Z)
• Knowledge Transfer by Discriminative Pre-training for Academic Performance Prediction [5.3431413737671525]
  We propose DPA, a transfer learning framework with Discriminative Pre-training tasks for Academic performance prediction. Compared to the previous state-of-the-art generative pre-training method, DPA is more sample efficient, leading to fast convergence to lower academic performance prediction error.
  arXiv  Detail & Related papers  (2021-06-28T13:02:23Z)
• Don't Stop Pretraining: Adapt Language Models to Domains and Tasks [81.99843216550306]
  We present a study across four domains (biomedical and computer science publications, news, and reviews) and eight classification tasks. A second phase of pretraining in-domain (domain-adaptive pretraining) leads to performance gains. Adapting to the task's unlabeled data (task-adaptive pretraining) improves performance even after domain-adaptive pretraining.
  arXiv  Detail & Related papers  (2020-04-23T04:21:19Z)
• Pre-training Text Representations as Meta Learning [113.3361289756749]
  We introduce a learning algorithm which directly optimize model's ability to learn text representations for effective learning of downstream tasks. We show that there is an intrinsic connection between multi-task pre-training and model-agnostic meta-learning with a sequence of meta-train steps.
  arXiv  Detail & Related papers  (2020-04-12T09:05:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.