CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Search

• URL: http://arxiv.org/abs/2412.01269v5
• Date: Tue, 18 Feb 2025 09:05:29 GMT
• Title: CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Search
• Authors: Kaixin Wu, Yixin Ji, Zeyuan Chen, Qiang Wang, Cunxiang Wang, Hong Liu, Baijun Ji, Jia Xu, Zhongyi Liu, Jinjie Gu, Yuan Zhou, Linjian Mo,
• Abstract summary: CPRM is a framework designed for the continual pre-training of large language models (LLMs) Our framework includes three modules: 1) employing both queries and multi-field item to jointly pre-train for enhancing domain knowledge, 2) applying in-context pre-training, and 3) conducting reading comprehension on items to produce associated domain knowledge and background information.
• Score: 34.08551439233784
• License:
• Abstract: Relevance modeling between queries and items stands as a pivotal component in commercial search engines, directly affecting the user experience. Given the remarkable achievements of large language models (LLMs) in various natural language processing (NLP) tasks, LLM-based relevance modeling is gradually being adopted within industrial search systems. Nevertheless, foundational LLMs lack domain-specific knowledge and do not fully exploit the potential of in-context learning. Furthermore, structured item text remains underutilized, and there is a shortage in the supply of corresponding queries and background knowledge. We thereby propose CPRM (Continual Pre-training for Relevance Modeling), a framework designed for the continual pre-training of LLMs to address these issues. Our CPRM framework includes three modules: 1) employing both queries and multi-field item to jointly pre-train for enhancing domain knowledge, 2) applying in-context pre-training, a novel approach where LLMs are pre-trained on a sequence of related queries or items, and 3) conducting reading comprehension on items to produce associated domain knowledge and background information (e.g., generating summaries and corresponding queries) to further strengthen LLMs. Results on offline experiments and online A/B testing demonstrate that our model achieves convincing performance compared to strong baselines.

Related papers

• Explainable LLM-driven Multi-dimensional Distillation for E-Commerce Relevance Learning [20.569157915157817]
  We propose an Explainable LLM-driven Multi-dimensional Distillation framework for e-commerce relevance learning. Our proposed framework significantly enhances e-commerce relevance learning performance and user experience.
  arXiv  Detail & Related papers  (2024-11-20T05:30:15Z)
• RuAG: Learned-rule-augmented Generation for Large Language Models [62.64389390179651]
  We propose a novel framework, RuAG, to automatically distill large volumes of offline data into interpretable first-order logic rules. We evaluate our framework on public and private industrial tasks, including natural language processing, time-series, decision-making, and industrial tasks.
  arXiv  Detail & Related papers  (2024-11-04T00:01:34Z)
• Towards Boosting LLMs-driven Relevance Modeling with Progressive Retrieved Behavior-augmented Prompting [23.61061000692023]
  This study proposes leveraging user interactions recorded in search logs to yield insights into users' implicit search intentions. We propose ProRBP, a novel Progressive Retrieved Behavior-augmented Prompting framework for integrating search scenario-oriented knowledge with Large Language Models.
  arXiv  Detail & Related papers  (2024-08-18T11:07:38Z)
• Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More? [54.667202878390526]
  Long-context language models (LCLMs) have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases. We introduce LOFT, a benchmark of real-world tasks requiring context up to millions of tokens designed to evaluate LCLMs' performance on in-context retrieval and reasoning. Our findings reveal LCLMs' surprising ability to rival state-of-the-art retrieval and RAG systems, despite never having been explicitly trained for these tasks.
  arXiv  Detail & Related papers  (2024-06-19T00:28:58Z)
• CRE-LLM: A Domain-Specific Chinese Relation Extraction Framework with Fine-tuned Large Language Model [1.9950682531209156]
  Domain-Specific Chinese Relation Extraction (DSCRE) aims to extract relations between entities from domain-specific Chinese text. Given the impressive performance of large language models (LLMs) in natural language processing, we propose a new framework called CRE-LLM.
  arXiv  Detail & Related papers  (2024-04-28T06:27:15Z)
• LLM Inference Unveiled: Survey and Roofline Model Insights [62.92811060490876]
  Large Language Model (LLM) inference is rapidly evolving, presenting a unique blend of opportunities and challenges. Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model. This framework identifies the bottlenecks when deploying LLMs on hardware devices and provides a clear understanding of practical problems.
  arXiv  Detail & Related papers  (2024-02-26T07:33:05Z)
• Generative Multi-Modal Knowledge Retrieval with Large Language Models [75.70313858231833]
  We propose an innovative end-to-end generative framework for multi-modal knowledge retrieval. Our framework takes advantage of the fact that large language models (LLMs) can effectively serve as virtual knowledge bases. We demonstrate significant improvements ranging from 3.0% to 14.6% across all evaluation metrics when compared to strong baselines.
  arXiv  Detail & Related papers  (2024-01-16T08:44:29Z)
• Knowledgeable In-Context Tuning: Exploring and Exploiting Factual Knowledge for In-Context Learning [37.22349652230841]
  Large language models (LLMs) enable in-context learning (ICL) by conditioning on a few labeled training examples as a text-based prompt. In this paper, we demonstrate that factual knowledge is imperative for the performance of ICL in three core facets. We introduce a novel Knowledgeable In-Context Tuning (KICT) framework to further improve the performance of ICL.
  arXiv  Detail & Related papers  (2023-09-26T09:06:39Z)
• Improving Open Information Extraction with Large Language Models: A Study on Demonstration Uncertainty [52.72790059506241]
  Open Information Extraction (OIE) task aims at extracting structured facts from unstructured text. Despite the potential of large language models (LLMs) like ChatGPT as a general task solver, they lag behind state-of-the-art (supervised) methods in OIE tasks.
  arXiv  Detail & Related papers  (2023-09-07T01:35:24Z)
• Synergistic Interplay between Search and Large Language Models for Information Retrieval [141.18083677333848]
  InteR allows RMs to expand knowledge in queries using LLM-generated knowledge collections. InteR achieves overall superior zero-shot retrieval performance compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-12T11:58:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.